Paper: arXiv 2511.15262
Authors: Tomas Espana, Yadh Hafsi, Fabrizio Lillo, Edoardo Vittori
Abstract
We investigate the use of Reinforcement Learning for the optimal execution of meta-orders, where the objective is to execute incrementally large orders while minimizing implementation shortfall and market impact over an extended period of time. Departing from traditional parametric approaches to price dynamics and impact modeling, we adopt a model-free, data-driven framework. Since policy optimization requires counterfactual feedback that historical data cannot provide, we employ the Queue-Reactive Model to generate realistic and tractable limit order book simulations that encompass transient price impact, and nonlinear and dynamic order flow responses. Methodologically, we train a Double Deep Q-Network agent on a state space comprising time, inventory, price, and depth variables, and evaluate its performance against established benchmarks. Numerical simulation results show that the agent learns a policy that is both strategic and tactical, adapting effectively to order book conditions and outperforming standard approaches across multiple training configurations. These findings provide strong evidence that model-free Reinforcement Learning can yield adaptive and robust solutions to the optimal execution problem.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 6.0/10 (capped from 8.0: no out-of-sample test; see penalty flags)
- Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)
Why this score: The paper employs advanced mathematical concepts like stochastic control theory, HJB equations, viscosity solutions, and transient impact models with arbitrage constraints, indicating high mathematical complexity. It also demonstrates strong empirical rigor through the use of a calibrated Queue-Reactive Model simulator for training and evaluation, a standard RL architecture (DDQN), and comparative benchmarking in simulated environments, making it backtest-ready.
Research Flowchart
flowchart TD Start["Research Goal:<br>Optimal Execution via RL"] --> Inputs["Key Inputs:<br>Limit Order Book Data"] Inputs --> Model["Methodology:<br>Queue-Reactive Model Simulation"] Model --> StateSpace["State Space Construction:<br>Time, Inventory, Price, Depth"] StateSpace --> DQN["Computational Process:<br>Train Double Deep Q-Network"] DQN --> Findings["Key Findings:<br>Adaptive Policy & Outperforms Benchmarks"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a reinforcement‑learning framework using a Double Deep Q‑Network to solve the optimal execution problem within a Queue‑Reactive Model of the limit order book.
Source sentence
In this work, we proposed a reinforcement‑learning framework for optimal execution within the Queue‑Reactive Model (QRM).
Assumes 5
- All order‑book queues are assumed non‑empty for simplicity. model
Source sentence
For simplicity, all queues q±i are considered non‑empty, as this represents the typical configuration of the invariant distribution π.
- The pre‑trade mid‑price equals the mid‑price just before the trade (pref(t‑k)=pmid(t‑k)). model
Source sentence
Furthermore, we assume pref(t−k)=pmid(t−k), with the surrounding volumes q±i sampled from the invariant distribution
- The execution price is always the best ask. execution
Source sentence
Pτk the execution price, which in our setting is always the best ask.
- A linear terminal penalty proportional to remaining shares is applied to enforce full execution. execution
Source sentence
This penalty is equal to the number of shares remaining to be executed, scaled by a final penalty parameter α
Show 1 more
- The underlying Queue‑Reactive process is ergodic under mild conditions. model
Source sentence
with the surrounding volumes q±i sampled from the invariant distribution, consistent with the ergodicity of the process X under mild conditions.
Data
- Queue‑Reactive Model simulations synthetic unknown
Source sentence
we employ the Queue‑Reactive Model to generate realistic and tractable limit order book simulations that encompass transient price impact, and nonlinear and dynamic order flow responses
Costs
Modeled
Implementation shortfall (IS) is used as the cost metric; the reward is defined as the negative of IS.
Source sentence
Since the reward function is defined as the negative of the IS, minimizing IS is equivalent to maximizing the reward.
Finds 4
- The learned policy consistently outperforms the best benchmark across different market regimes, achieving up to 27% higher performance. performance
Source sentence
it consistently exceeds the best benchmark, achieving up to 27% higher performance.
- Feature‑gradient analysis shows that ask price and inventory are the primary determinants of action selection, while time has a small influence. other
Source sentence
The results indicate that the ask price and inventory are the primary determinants of action selection, suggesting that the agent mainly reacts to immediate market conditions.
- Q‑values decline over time for a fixed inventory, reflecting increasing risk of incomplete execution as the horizon shortens. other
Source sentence
All plots reveal that, for a fixed inventory, Q‑values decline over time as the remaining horizon shortens, reflecting the agent’s increasing risk of incomplete execution
- The agent learns a policy that is both strategic and tactical, adapting effectively to order‑book conditions and outperforming standard approaches across multiple training configurations. performance
Source sentence
Numerical simulation results show that the agent learns a policy that is both strategic and tactical, adapting effectively to order book conditions and outperforming standard approaches across multiple training configurations.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 36 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (151)
- Optimal execution in a time varying environment: well posedness and price manipulation (2024)
- Markov decision processes: discrete stochastic dynamic programming (2014)
- Human-level control through deep reinforcement learning (2015)
- Dynamic programming (1966)
- Optimal signal-adaptive trading with temporary and transient price impact (2020) — link
- Optimal trading with stochastic liquidity and volatility (2012)
- Optimal execution strategies in limit order books with general shape functions (2010)
- Optimal trade execution and absence of price manipulations in limit order book models (2010)
and 143 more
- Stochastic duration and fast coupon bond option pricing in multi-factor models (1999)
- An exact bond option pricing formula (1989)
- Portfolio liquidation and ambiguity aversion (2018)
- Portfolio liquidation under factor uncertainty (2022)
- Optimal execution: A review (2022)
- Robust strategies for optimal order execution in the Almgren–Chriss framework (2013)
- V-shapes (2020)
- Transient linear price impact and Fredholm integral equations (2012)
- Optimal portfolio liquidation with limit orders (2012)
- Liquidation in limit order books with controlled intensity (2014)
- Optimal execution of portfolio transactions (2001)
- Optimal trading strategy and supply/demand dynamics (2013)
- The Pricing of Commodity Contracts. (1976)
- Theory of The Term Structure of Interest Rates (1985)
- Hedging under SABR model (2006)
- Fine-tune your smile: Correction to Hagan et al. (2008)
- The pricing of options and corporate liabilities (1973)
- An equilibrium characterization of the term structure (1977)
- Everything You Always Wanted to Know About Multiple Interest Rate Curve Bootstrapping but Were Afraid to Ask (2016)
- Impact of multiple curve dynamics in credit valuation adjustments under collateralization (2018)
- Price Impact on Term Structure (2020)
- Optimal dealer pricing under transaction and return uncertainty (1981)
- The dynamics of dealer markets under competition (1983)
- Inferring the components of the bid ask spread (1989)
- Bid, ask and transaction prices in a specialist market with heterogeneously informed traders (1985)
- Optimal portfolio liquidation with execution cost and risk (2010)
- Interest rates and the credit crunch: new formulas and market models (2009)
- Incorporating signals into optimal trading (2019)
- Interest-rate modelling in collateralized markets: multiple curves, credit-liquidity effects, CCPs (2013) — link
- Moving from IBORs to Alternative Risk Free Rates (2020)
- Almost-sure hedging with permanent price impact (2016)
- Pricing and hedging contingent claims with liquidity costs and market impact (2013)
- Optimal trade execution under displaced diffusions dynamics across different risk criteria (2014)
- Optimal trade execution: a mean quadratic variation approach (2012)
- Comparison between the mean-variance optimal and the mean-quadratic-variation optimal trading strategies (2013)
- General intensity shapes in optimal liquidation (2015)
- Incorporating order-flow into optimal execution (2016)
- Trading algorithms with learning in latent alpha models (2019)
- Q-learning (1992)
- Playing atari with deep reinforcement learning (2013) — link
- Deep reinforcement learning with double q-learning (2016)
- Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019)
- Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks (2019)
- Reinforcement learning for quantitative trading (2023)
- Recent advances in reinforcement learning in finance (2021) — link
- No-dynamic-arbitrage and market impact (2010)
- Price manipulation and quasi-arbitrage (2004)
- Adaptive market making via online learning (2013)
- Deep reinforcement learning for market making (2020)
- Model-based deep reinforcement learning for dynamic portfolio optimization (2019) — link
- Machine learning for market microstructure and high frequency trading (2013)
- Model-based reinforcement learning for predictions and control for limit order books (2019) — link
- Deep reinforcement learning for active high frequency trading (2021) — link
- Multi-agent reinforcement learning in a realistic limit order book market simulation (2020)
- Deep reinforcement learning for the optimal placement of cryptocurrency limit orders (2022)
- Continuous auctions and insider trading (1985)
- Deep differentiable reinforcement learning and optimal trading (2022)
- Optimal trading with signals and stochastic price impact (2022)
- Liquidity fluctuations and the latent dynamics of price impact (2022)
- Optimal control of execution costs (1998)
- Direct estimation of equity market impact (2005)
- Fluctuations and response in financial markets: the subtle nature ofrandom’price changes (2003)
- How Markets Slowly Digest Changes in Supply and Demand (2009)
- Optimal basket liquidation for CARA investors is deterministic (2010)
- Algorithmic and high-frequency trading (2015)
- Optimal liquidation under stochastic price impact (2019)
- Measuring price impact and information content of trades in a time-varying setting — → 2212.12687
- A unified approach to interpreting model predictions (2017)
- The cost of misspecifying price impact — → 2306.00599
- Double Q-learning (2010)
- Issues in using function approximation for reinforcement learning (2014)
- Learning from delayed rewards (1989)
- A reinforcement learning approach to optimal execution (2021) — link
- Improving reinforcement learning algorithms: Towards optimal learning rate policies (2024)
- Thinking fast and slow with deep learning and tree search (2017)
- Identifying new classes of financial price jumps with wavelets (2025)
- Generative ai for end-to-end limit order book modelling: A token-level autoregressive generative model of message flow using a deep state space network — → 2309.00638
- LOB-Bench: Benchmarking Generative AI for Finance–an Application to Limit Order Book Data — → 2502.09172
- Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019) — link
- Deep Execution - Value and Policy Based Reinforcement Learning for Trading and Beating Market Benchmarks (2019) — link
- Reinforcement learning for optimal execution when liquidity is time-varying — → 2402.12049
- Optimal execution with reinforcement learning — → 2411.06389
- Reinforcement Learning for Trade Execution with Market Impact — → 2507.06345
- Learning a functional control for high-frequency finance (2022) — link
- Multi-agent reinforcement learning in a realistic limit order book market simulation (2020) — link
- Double deep q-learning for optimal execution (2021)
- A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
- Reinforcement learning: An introduction (1998)
- Optimal execution with stochastic delay (2023)
- Optimal order execution under price impact: a hybrid model (2024)
- Optimal execution with non-linear transient market impact (2017)
- Optimal execution with limit and market orders (2015)
- Optimal trade execution under stochastic volatility and liquidity (2014)
- Optimal trade execution under geometric Brownian motion in the Almgren and Chriss framework (2011)
- Optimal execution with nonlinear impact functions and trading-enhanced risk (2003)
- Linear Models for the Impact of Order Flow on Prices I. Propagators: Transient vs. History Dependent Impact (2016) — link
- Crossover from Linear to Square-Root Market Impact (2018) — link
- Market efficiency and the long-memory of supply and demand: is price impact variable and permanent or fixed and temporary? (2006) — link
- Trades, Quotes and Prices: Financial Markets Under the Microscope (2018)
- Handbook of price impact modeling (2023)
- Multi-agent reinforcement learning in a realistic limit order book market simulation (2020) — link
- Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets — → 2301.08688
- How to Evaluate Trading Strategies: Single Agent Market Replay or Multiple Agent Interactive Simulation? (2019)
- Proximal policy optimization algorithms (2017) — link
- An end-to-end optimal trade execution framework based on proximal policy optimization (2021)
- Human-level control through deep reinforcement learning (2015)
- Reinforcement learning for optimized trade execution (2006)
- Simulating and analyzing order book data: The queue-reactive model (2015)
- Deep reinforcement learning with double q-learning (2016)
- How markets slowly digest changes in supply and demand (2009) — link
- The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making (2016) — link
- Optimal trading with stochastic liquidity and volatility (2012)
- Algorithmic Trading, Stochastic Control, and Mutually-Exciting Processes (2018) — link
- Dynamic Programming and Optimal Control, Volume I (2005)
- Quasivariational Inequalities in Stochastic Control: The Continuous Case (2002)
- Applied Stochastic Control of Jump Diffusions (2007)
- Backward Stochastic Differential Equations with Constraints on the Terminal Value (1997)
- Viscosity solutions of Hamilton-Jacobi equations (1983) — link
- Monte Carlo Methods in Financial Engineering (2004)
- No-dynamic-arbitrage and market impact (2010) — link
- Anomalous price impact and the critical nature of liquidity in financial markets (2011)
- How efficiency shapes market impact (2013) — link
- Ergodicity and diffusivity of Markovian order book models: a general framework (2017)
- From zero-intelligence to queue-reactive: limit-order-book modeling for high-frequency volatility estimation and optimal execution (2022) — link
- Queue-reactive Hawkes models for the order flow (2021)
- A Novel Approach to Queue-Reactive Models: The Importance of Order Sizes — → 2405.18594
- Deep Learning Meets Queue-Reactive: A Framework for Realistic Limit Order Book Simulation — → 2501.08822
- Limit Order Book Simulation with Generative Adversarial Networks (2023) — link
- Generating realistic stock market order streams (2020)
- Learning to simulate realistic limit order book markets from data as a world agent (2022)
- ABIDES: Towards high-fidelity multi-agent market simulation (2020)
- Minimal agent based model for financial markets II: statistical properties of the linear and multiplicative dynamics (2009)
- Econophysics review: II. Agent-based models (2011)
- Agent-based modelling in economics (2015)
- Optimal liquidity trading (2005)
- Optimal execution under incomplete information — → 2411.04616
- Optimal Execution under Liquidity Uncertainty — → 2506.11813
- Multiple-limit trades: empirical facts and application to lead–lag measures (2013)
- Master curve for price-impact function (2003) — link
- Understanding the worst-kept secret of high-frequency trading — → 2307.15599
- The implementation shortfall: Paper versus reality (1988)
- Asset pricing under endogenous expectations in an artificial stock market (1997)
- Linear models for the impact of order flow on prices. I. History dependent impact models (2018)
13 of 151 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.