{"paper_id":"2411.06389","title":"Optimal Execution with Reinforcement Learning","authors":["Yadh Hafsi","Edoardo Vittori"],"paper_date":"2024-11-10","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2411.06389v2","page_url":"https://thequant.space/flowcharts/optimal-execution-with-reinforcement-learning/","scores":{"empirical_rigor":8,"math_complexity":7.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a reinforcement learning framework using a Double Q Network (DQN) trained on a multi‑agent market simulator to learn an optimal execution strategy.","quote":"In this paper we present a framework based on RL, specifically the Double Q Network (DQN) [12] algorithm, to learn an optimal execution strategy.","summary":"This study investigates the development of an optimal execution strategy through reinforcement learning, aiming to determine the most effective approach for traders to buy and sell inventory within a finite time horizon. Our proposed model leverages input features derived from the current state of…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The filtration supporting the simulation contains a standard Brownian motion and a Poisson process with intensity λ.","type":"model","stated":"explicit","quote":"We assume that the filtration {Ft}t≥0 supports a standard P‑Brownian motion W and a random Poisson process N with intensity λ."},{"assumption":"The fundamental value follows an Ornstein‑Uhlenbeck process with jumps representing news events.","type":"model","stated":"explicit","quote":"The dynamics of the fundamental process process can be described by the following stochastic differential equation (SDE): dXt = θ(µ −Xt) dt + σ dWt + J dNt,"},{"assumption":"Training and testing are performed on synthetic data generated by the ABIDES multi‑agent market simulator.","type":"data","stated":"explicit","quote":"Training and testing occur on synthetic data generated through the multi‑agent market simulator ABIDES."},{"assumption":"The agent operates over a 30‑minute horizon with one‑second control intervals.","type":"execution","stated":"explicit","quote":"By training a reinforcement learning agent to operate over a 30‑minute window with one‑second control intervals"},{"assumption":"The ABIDES RMSC‑4 configuration accurately reproduces realistic market impact.","type":"market","stated":"explicit","quote":"We chose the RMSC‑4 configuration, which is the reference configuration in ABIDES"},{"assumption":"Implementation shortfall (IS) is an appropriate metric for evaluating execution performance.","type":"other","stated":"explicit","quote":"We test whether RL outperforms the benchmark strategies on implementation shortfall (IS) using one‑sided Student’s two‑sample t‑tests"}]},"datasets":{"status":"available","items":[{"name":"ABIDES multi‑agent market simulator (RMSC‑4 configuration)","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"Training and testing occur on synthetic data generated through the multi‑agent market simulator ABIDES.","known_id":"abides"}],"known":[{"id":"abides","name":"ABIDES simulator","provider":"ABIDES","kind":"synthetic","access":"public","quote":"To simulate this environment and overcome the limitations associated with relying on historical data, we utilize the multi-agent market simulator ABIDES, which provides a diverse range of depth levels within the limit order book.","mentions":8,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":"Execution performance is evaluated using implementation shortfall (IS), which captures market impact and transaction costs.","quote":"We test whether RL outperforms the benchmark strategies on implementation shortfall (IS) using one‑sided Student’s two‑sample t‑tests","source":"front_matter","rule":"R3","candidates":{"front_matter":"yes","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Execution performance is evaluated using implementation shortfall (IS), which captures market impact and transaction costs.","card_quote":"We test whether RL outperforms the benchmark strategies on implementation shortfall (IS) using one‑sided Student’s two‑sample t‑tests","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"execution-cost","findings":[{"finding":"RL significantly outperforms TWAP and Passive benchmarks with 10 noise agents (t = 4.52 for TWAP, t = 2.08 for Passive).","kind":"performance","quote":"With 10 noise agents, RL significantly outperforms TWAP (t = 4.52) and Passive (t = 2.08)."},{"finding":"RL significantly outperforms all benchmarks (TWAP, Random, Passive) with 2000 noise agents.","kind":"performance","quote":"With 2000 noise agents, RL significantly outperforms all benchmarks: TWAP (t = 4.75), Random (t = 8.49), and Passive (t = 2.42)."},{"finding":"RL consistently achieves statistically significant gains across baseline, noise‑agent stress, and momentum‑driven environments.","kind":"performance","quote":"Overall, RL consistently achieves statistically significant gains in the baseline and under noise‑agent stress, and also demonstrates advantages in momentum‑driven environments."},{"finding":"The RL agent attains higher returns and lower variance in implementation shortfall compared to benchmark strategies.","kind":"performance","quote":"Using the DQN algorithm within a simulated market environment, we observed significant improvements in execution performance compared to the chosen benchmark strategies."},{"finding":"The RL agent’s execution trajectory balances immediate market impact with long‑term price stability, aligning with optimal execution principles.","kind":"theoretical","quote":"The execution trajectory analysis highlighted the strategic balance of the RL agent between immediate market impact and long‑term price stability."}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":21,"citations":{"cited_by_in_corpus":5,"by_independent_authors":4,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":1,"mentions":3,"unclassified":1},"independent_papers":[{"paper":{"paper_id":"2510.22206","title":"Right Place, Right Time: Market Simulation-based RL for Execution Optimisation","paper_date":"2025-10-25","authors":["Ollie Olby","Andreea Bacalum","Rory Baggott","Namid Stillman"],"empirical_rigor":7,"page_url":"https://thequant.space/flowcharts/right-place-right-time-market-simulation-based-rl-for-exec/"},"how":"discusses_findings","quote":"Their results showed that the RL agent outperforms traditional strategies such as TWAP, particularly in terms of execution consistency and reduced market impact."}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T16:29:33Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}