Paper: arXiv 2403.19781

Authors: Zhiyuan Yao, Zheng Li, Matthew Thomas, Ionut Florescu

Abstract

Investors and regulators can greatly benefit from a realistic market simulator that enables them to anticipate the consequences of their decisions in real markets. However, traditional rule-based market simulators often fall short in accurately capturing the dynamic behavior of market participants, particularly in response to external market impact events or changes in the behavior of other participants. In this study, we explore an agent-based simulation framework employing reinforcement learning (RL) agents. We present the implementation details of these RL agents and demonstrate that the simulated market exhibits realistic stylized facts observed in real-world markets. Furthermore, we investigate the behavior of RL agents when confronted with external market impacts, such as a flash crash. Our findings shed light on the effectiveness and adaptability of RL-based agents within the simulation, offering insights into their response to significant market events.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper employs advanced mathematical concepts like Markov Decision Processes (MDP) and the Proximal Policy Optimization (PPO) algorithm, evidenced by detailed formal definitions and equations, which drives the high math complexity score. However, the empirical rigor is limited as the study is framed within a simulation environment using synthetic agents rather than backtesting on real financial data, with no mention of live deployment or statistical validation against market outcomes.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Develop an RL-based Agent-Based<br>Market Simulator to capture<br>dynamic market behavior &<br>realistic stylized facts"] --> B

  subgraph B ["Methodology"]
      direction TB
      B1["Define RL Agent<br>Architecture & Parameters"] --> B2["Train RL Agents<br>in Market Environment"] --> B3["Deploy Trained Agents<br>into Agent-Based Model"]
  end

  B --> C

  subgraph C ["Data & Inputs"]
      C1["Market Data for<br>Environment State"]
      C2["Reward Functions &<br>Trading Rules"]
  end

  C --> D["Computational Process:<br>Multi-Agent Simulation<br>with RL Decision Logic"]

  D --> E

  subgraph E ["Key Findings/Outcomes"]
      direction TB
      E1["Simulated market exhibits<br>realistic stylized facts<br>e.g., volatility clustering, fat tails"]
      E2["RL agents adapt dynamically<br>to external shocks e.g., Flash Crash"]
      E3["Framework offers effective<br>tool for anticipating policy<br>& regulatory impacts"]
  end

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes an agent‑based market simulation framework that employs reinforcement‑learning agents to generate realistic market dynamics and to study their behavior under external impacts such as flash crashes.

Source sentence
In this study, we explore an agent-based simulation framework employing reinforcement learning (RL) agents. We present the implementation details of these RL agents and demonstrate that the simulated market exhibits realistic stylized facts observed in real-world markets.

Assumes 6

  • RL agents are trained assuming unchanged system dynamics during a simulation. model
    Source sentence
    In general, RL agents assume unchanged dynamics of the system.
  • In real markets the system dynamics evolve when interacting with other traders, which the study tests by allowing training to persist. market
    Source sentence
    However, in real-world scenarios when interacting with other traders and in a dynamic market, the system dynamics evolve.
  • The ECN component used in the original formulation is omitted in the current simulation. model
    Source sentence
    In our simulation, the ECN is gone.
  • Random network latency is introduced to add true randomness to the simulation. execution
    Source sentence
    Additionally, random latency due to network communication introduces an element of true randomness to the simulation, enhancing its realism.
Show 2 more
  • The simulation environment is closed with no outside information unless an external impact (flash sale) is deliberately introduced. market
    Source sentence
    4.1.2 Market Responsiveness A simulated market within a closed environment where trading is driven by traders without any outside information should be stable. However, we are interested in how the simulation will behave when we additional external impact is introduced.
  • Agents’ reward functions define their investment goals and drive their learning behavior. model
    Source sentence
    The other class of RL agents are liquidity takers which have an investment goal defined by their own reward function.

Data

  • LOBSTER limit order book data · LOBSTER order-book public
    2012-06-21 · Google, Apple, Amazon
    Source sentence
    5 Experiment Results 5.1 Statistical Characteristics of Observed Asset Prices We use Google, Apple, and Amazon’s tick level limit order book data on 2012-06-21 from LOBSTER [28].
  • Zero‑Intelligence (ZI) agent simulation synthetic unknown
    Source sentence
    Figure 2: Illustration of Groups for Comparison To compare the results produced by the RL agents we introduce an additional simulation model using 100 Zero-Intelligence (ZI) agents. This system is using the agent design in Farmer et al.’s work [9] and [22].
  • RL‑agent simulation (continual learning Group A) synthetic unknown
    Source sentence
    Additionally, we investigate the evolution of Market-Maker (MM) agents’ inventory and PnL components across different groups. To assess responsiveness, we introduce a sequence of flash sale events, and we examine the price impact during the flash sale period.

Recognised sources: LOBSTER

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • Simulated markets using RL agents reproduce realistic stylized facts such as heavy‑tailed return distributions and slow decay of absolute‑return autocorrelation, closely matching real data. statistical
    Source sentence
    The real data, as well as data obtained using RL and ZI agents, all exhibit strong fat tail return and price distributions. Results from RL agents closely align with the real data.
  • Continual‑learning RL agents generate the most realistic market simulations, outperforming both static RL agents and ZI agents. performance
    Source sentence
    We find that continual learning RL agents produce the most realistic market simulation, and are capable of adapting to changing market conditions.
  • During flash‑sale experiments, untrained agents cause a permanent price drop, while continually trained agents exhibit temporary and permanent price impacts consistent with established market impact literature. statistical
    Source sentence
    The untrained group shows a significant and permanent price drop in contrast to the testing and continual training groups. However, the continual training group shows more realistic trajectories, with prices dropping and recovering to levels lower than the pre-sale prices.
  • Continual‑learning agents adapt their order‑placement tweaks, placing deeper orders and lowering expected price equilibrium after flash‑sale detection. other
    Source sentence
    However, when fed imbalanced LOBs (Figures 8a and 8c), the agents in the continual training group output higher symmetric price tweaks and lower asymmetric price tweaks, compared with the agents in the two other groups. Thus when a flash sale event is detected, the MM agents in the continual learning group tend to place orders deeper into the LOB and move the expected price equilibrium downward.
  • When informed LT agents change buy/sell preferences, MM agents become more conservative (larger symmetric tweak) and adjust asymmetric tweaks in line with price direction, mirroring earlier experiment results. other
    Source sentence
    In both scenarios when the price is going down and when the price is going up, the action symmetric tweak (top row) gets larger, thus the MM agent becomes more conservative and tends to enlarge the spread. This aligns with the findings in the previous experiment.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 24 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 3 papers in the archive (3 by independent authors): Right Place, Right Time: Market Simulation-based RL for … (uses its method or data)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (30)

  1. Towards a fully RL-based Market Simulator (2021)
  2. SHIFT: A Highly Realistic Financial Market Simulation Platform (2020)
  3. Modelling stock markets by multi-agent reinforcement learning (2021)
  4. More stylized facts of financial markets: leverage effect and downside correlations (2001)
  5. Empirical properties of asset returns: stylized facts and statistical issues (2001)
  6. A stochastic model for order book dynamics (2010)
  7. Proximal policy optimization algorithms (2017) — link
  8. Deep reinforcement learning in agent based financial market simulation (2020)
and 22 more
  1. Agent-based simulation of a financial market (2001)
  2. Multi-agent simulation of financial markets (2005)
  3. Learning to simulate realistic limit order book markets from data as a World Agent (2022)
  4. A dynamic model of the limit order book (2009)
  5. Revisiting Stylized Facts for Modern Stock Markets (2023)
  6. Optimal execution of portfolio transactions (2001)
  7. Statistical theory of the continuous double auction (2003)
  8. Reinforcement learning: An introduction (2018)
  9. TradeMaster: A Holistic Quantitative Trading Platform Empowered by Reinforcement Learning (2023)
  10. Credit rating by hybrid machine learning techniques (2010)
  11. A sparsity algorithm for finding optimal counterfactual explanations: Application to corporate credit rating (2023)
  12. A reinforcement learning approach to optimal execution (2022)
  13. Reinforcement learning for optimized trade execution (2006)
  14. Performance functions and reinforcement learning for trading systems and portfolios (1998)
  15. Strategic agent-based modeling of financial markets (2017)
  16. Human-level control through deep reinforcement learning (2015)
  17. Reinforcement learning and optimal control (2019)
  18. The predictive power of zero intelligence in financial markets (2005)
  19. Describing the dynamic nature of transactions costs during political event risk episodes (2018) — link
  20. Information effects on the bid-ask spread (1983)
  21. Bid, ask and transaction prices in a specialist market with heterogeneously informed traders (1985)
  22. Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations (2022) — link

0 of 30 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.