{"paper_id":"2403.19781","title":"Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior","authors":["Zhiyuan Yao","Zheng Li","Matthew Thomas","Ionut Florescu"],"paper_date":"2024-03-28","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2403.19781v1","page_url":"https://thequant.space/flowcharts/reinforcement-learning-in-agent-based-market-simulation-unv/","scores":{"empirical_rigor":3,"math_complexity":7.5,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes an agent‑based market simulation framework that employs reinforcement‑learning agents to generate realistic market dynamics and to study their behavior under external impacts such as flash crashes.","quote":"In this study, we explore an agent-based simulation framework employing reinforcement learning (RL) agents. We present the implementation details of these RL agents and demonstrate that the simulated market exhibits realistic stylized facts observed in real-world markets.","summary":"Investors and regulators can greatly benefit from a realistic market simulator that enables them to anticipate the consequences of their decisions in real markets. However, traditional rule-based market simulators often fall short in accurately capturing the dynamic behavior of market participants…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"RL agents are trained assuming unchanged system dynamics during a simulation.","type":"model","stated":"explicit","quote":"In general, RL agents assume unchanged dynamics of the system.","support":"full"},{"assumption":"In real markets the system dynamics evolve when interacting with other traders, which the study tests by allowing training to persist.","type":"market","stated":"explicit","quote":"However, in real-world scenarios when interacting with other traders and in a dynamic market, the system dynamics evolve.","support":"full"},{"assumption":"The ECN component used in the original formulation is omitted in the current simulation.","type":"model","stated":"explicit","quote":"In our simulation, the ECN is gone.","support":"full"},{"assumption":"Random network latency is introduced to add true randomness to the simulation.","type":"execution","stated":"explicit","quote":"Additionally, random latency due to network communication introduces an element of true randomness to the simulation, enhancing its realism.","support":"full"},{"assumption":"The simulation environment is closed with no outside information unless an external impact (flash sale) is deliberately introduced.","type":"market","stated":"explicit","quote":"4.1.2 Market Responsiveness A simulated market within a closed environment where trading is driven by traders without any outside information should be stable. However, we are interested in how the simulation will behave when we additional external impact is introduced.","support":"full"},{"assumption":"Agents’ reward functions define their investment goals and drive their learning behavior.","type":"model","stated":"explicit","quote":"The other class of RL agents are liquidity takers which have an investment goal defined by their own reward function.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"LOBSTER limit order book data","provider":"LOBSTER","kind":"order-book","access":"public","period":"2012-06-21","universe":"Google, Apple, Amazon","quote":"5 Experiment Results 5.1 Statistical Characteristics of Observed Asset Prices We use Google, Apple, and Amazon’s tick level limit order book data on 2012-06-21 from LOBSTER [28].","support":"full","known_id":"lobster"},{"name":"Zero‑Intelligence (ZI) agent simulation","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"Figure 2: Illustration of Groups for Comparison To compare the results produced by the RL agents we introduce an additional simulation model using 100 Zero-Intelligence (ZI) agents. This system is using the agent design in Farmer et al.’s work [9] and [22].","support":"full"},{"name":"RL‑agent simulation (continual learning Group A)","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"Additionally, we investigate the evolution of Market-Maker (MM) agents’ inventory and PnL components across different groups. To assess responsiveness, we introduce a sequence of flash sale events, and we examine the price impact during the flash sale period.","support":"full"}],"known":[{"id":"lobster","name":"LOBSTER","provider":"LOBSTER","kind":"order-book","access":"commercial","quote":"5 Experiment Results 5.1 Statistical Characteristics of Observed Asset Prices We use Google, Apple, and Amazon’s tick level limit order book data on 2012-06-21 from LOBSTER [28].","mentions":1,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"l2-book"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"n/a","evidence_card":"no","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"Simulated markets using RL agents reproduce realistic stylized facts such as heavy‑tailed return distributions and slow decay of absolute‑return autocorrelation, closely matching real data.","kind":"statistical","quote":"The real data, as well as data obtained using RL and ZI agents, all exhibit strong fat tail return and price distributions. Results from RL agents closely align with the real data.","support":"full"},{"finding":"Continual‑learning RL agents generate the most realistic market simulations, outperforming both static RL agents and ZI agents.","kind":"performance","quote":"We find that continual learning RL agents produce the most realistic market simulation, and are capable of adapting to changing market conditions.","support":"full"},{"finding":"During flash‑sale experiments, untrained agents cause a permanent price drop, while continually trained agents exhibit temporary and permanent price impacts consistent with established market impact literature.","kind":"statistical","quote":"The untrained group shows a significant and permanent price drop in contrast to the testing and continual training groups. However, the continual training group shows more realistic trajectories, with prices dropping and recovering to levels lower than the pre-sale prices.","support":"full"},{"finding":"Continual‑learning agents adapt their order‑placement tweaks, placing deeper orders and lowering expected price equilibrium after flash‑sale detection.","kind":"other","quote":"However, when fed imbalanced LOBs (Figures 8a and 8c), the agents in the continual training group output higher symmetric price tweaks and lower asymmetric price tweaks, compared with the agents in the two other groups. Thus when a flash sale event is detected, the MM agents in the continual learning group tend to place orders deeper into the LOB and move the expected price equilibrium downward.","support":"full"},{"finding":"When informed LT agents change buy/sell preferences, MM agents become more conservative (larger symmetric tweak) and adjust asymmetric tweaks in line with price direction, mirroring earlier experiment results.","kind":"other","quote":"In both scenarios when the price is going down and when the price is going up, the action symmetric tweak (top row) gets larger, thus the MM agent becomes more conservative and tends to enlarge the spread. This aligns with the findings in the previous experiment.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":24,"citations":{"cited_by_in_corpus":3,"by_independent_authors":3,"how_cited":{"used_as_baseline":0,"uses_method_or_data":1,"extends":0,"discusses_findings":0,"mentions":2,"unclassified":0},"independent_papers":[{"paper":{"paper_id":"2510.22206","title":"Right Place, Right Time: Market Simulation-based RL for Execution Optimisation","paper_date":"2025-10-25","authors":["Ollie Olby","Andreea Bacalum","Rory Baggott","Namid Stillman"],"empirical_rigor":7,"page_url":"https://thequant.space/flowcharts/right-place-right-time-market-simulation-based-rl-for-exec/"},"how":"uses_method_or_data","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:58:13Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}