Paper: arXiv 2402.12049

Authors: Andrea Macrì, Fabrizio Lillo

Abstract

Optimal execution is an important problem faced by any trader. Most solutions are based on the assumption of constant market impact, while liquidity is known to be dynamic. Moreover, models with time-varying liquidity typically assume that it is observable, despite the fact that, in reality, it is latent and hard to measure in real time. In this paper we show that the use of Double Deep Q-learning, a form of Reinforcement Learning based on neural networks, is able to learn optimal trading policies when liquidity is time-varying. Specifically, we consider an Almgren-Chriss framework with temporary and permanent impact parameters following several deterministic and stochastic dynamics. Using extensive numerical experiments, we show that the trained algorithm learns the optimal policy when the analytical solution is available, and overcomes benchmarks and approximated solutions when the solution is not available.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper employs advanced mathematical formulations, including stochastic differential equations and a Double Deep Q-learning neural network architecture, reflecting high mathematical density. However, the empirical validation relies on simulated data and numerical experiments rather than live trading, backtesting on real historical data, or public code repositories, resulting in lower empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Optimal Execution with<br>Time-Varying Liquidity?"] --> B["Methodology<br>Double Deep Q-Learning RL"]
  B --> C["Data & Framework<br>Almgren-Chriss Model<br>Stochastic Impact Parameters"]
  C --> D["Computational Process<br>Neural Network Training<br>Simulation & Learning"]
  D --> E["Key Findings<br>1. Matches analytical solutions<br>2. Outperforms benchmarks<br>3. Handles latent liquidity"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a Double Deep Q‑Learning (DDQL) algorithm that learns optimal liquidation strategies under time‑varying liquidity.

Source sentence
we show that the use of Double Deep Q-learning, a form of Reinforcement Learning based on neural networks, is able to learn optimal trading policies when liquidity is time-varying.

Assumes 5

  • The trader must unwind an initial inventory q0 within a fixed time window [0,T] divided into N equal sub‑intervals. execution
    Source sentence
    We will consider the problem of a trader who wants to unwind a portfolio of q0 stocks within a time window [0,T].
  • The impact of trades on price follows the Almgren‑Chriss linear market impact model. model
    Source sentence
    The Almgren and Chriss [3] (A&C) model is used as the baseline model.
  • The agent is risk‑neutral (risk‑aversion parameter λ = 0), implying a TWAP strategy in the baseline. model
    Source sentence
    In this paper the trader (or agent) is assumed to be risk neutral with λ = 0, thus a risk‑neutral agent in discrete time adopts the TWAP strategy.
  • Permanent and temporary impact coefficients can be deterministic or stochastic functions of time. market
    Source sentence
    We consider a setting where liquidity, as described by the temporary and permanent impact coefficient, are not constant but time varying.
Show 1 more
  • Liquidity (impact coefficients) is latent and not directly observable by the algorithm. other
    Source sentence
    models with time‑varying liquidity typically assume that it is observable, despite the fact that, in reality, it is latent and hard to measure in real time.

Data

Synthetic data only.

Costs

Modeled Costs are part of the reported headline results.

Finds 5

  • The DDQL agent learns the optimal policy when an analytical solution is available, matching the benchmark costs. theoretical
    Source sentence
    Using extensive numerical experiments, we show that the trained algorithm learns the optimal policy when the analytical solution is available
  • In the constant‑impact setting, DDQL achieves ΔP&L of –0.455 bps (features Q,T) and –0.225 bps (features Q,T,S), comparable to the Almgren‑Chriss model. performance
    Source sentence
    Table 2: Constant impact. Comparison between the costs of the DDQL agent and of the A&C model. ΔP&L values are reported in basis points.
  • With stochastic low‑λ impact, DDQL with features Q,T,S attains an average IS of 0.2572 and std.dev. 0.007, improving over the baseline. performance
    Source sentence
    Table 10: Stochastic impact. Comparison between DDQL agent and theoretical solutions. ΔP&L values are reported in basis points.
  • Under stochastic high‑λ impact, DDQL with features Q,T,S reduces average IS to 0.3158 (std.dev. 0.0112) versus the theoretical 0.5017, showing superior performance. performance
    Source sentence
    Table 11: Stochastic impact. Comparison between DDQL agent and theoretical solutions. ΔP&L values are reported in basis points.
  • When only inventory and time are provided as features, the DDQL agent reproduces a TWAP‑like strategy with ΔP&L of –0.4 bps and low variance. performance
    Source sentence
    The ΔP&L shows a difference of just −0.4 basis points and a standard deviation of only two basis points.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 19 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 8 papers in the archive (6 by independent authors): Mixture-of-Experts for Cryptocurrency Order Execution: … (uses its method or data)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (79)

  1. Optimal execution in a time varying environment: well posedness and price manipulation (2024)
  2. Optimal signal-adaptive trading with temporary and transient price impact (2020) — link
  3. Optimal trading with stochastic liquidity and volatility (2012)
  4. Optimal execution strategies in limit order books with general shape functions (2010)
  5. Optimal trade execution and absence of price manipulations in limit order book models (2010)
  6. Stochastic duration and fast coupon bond option pricing in multi-factor models (1999)
  7. An exact bond option pricing formula (1989)
  8. Portfolio liquidation and ambiguity aversion (2018)
and 71 more
  1. Portfolio liquidation under factor uncertainty (2022)
  2. Optimal execution: A review (2022)
  3. Robust strategies for optimal order execution in the Almgren–Chriss framework (2013)
  4. V-shapes (2020)
  5. Optimal execution with non-linear transient market impact (2017)
  6. Transient linear price impact and Fredholm integral equations (2012)
  7. Optimal portfolio liquidation with limit orders (2012)
  8. Liquidation in limit order books with controlled intensity (2014)
  9. Optimal execution of portfolio transactions (2000)
  10. Optimal control of execution costs (1998)
  11. Optimal trading strategy and supply/demand dynamics (2013)
  12. The Pricing of Commodity Contracts. (1976)
  13. Theory of The Term Structure of Interest Rates (1985)
  14. Hedging under SABR model (2006)
  15. Fine-tune your smile: Correction to Hagan et al. (2008)
  16. The pricing of options and corporate liabilities (1973)
  17. An equilibrium characterization of the term structure (1977)
  18. Everything You Always Wanted to Know About Multiple Interest Rate Curve Bootstrapping but Were Afraid to Ask (2016)
  19. Impact of multiple curve dynamics in credit valuation adjustments under collateralization (2018)
  20. Price Impact on Term Structure (2020)
  21. Optimal dealer pricing under transaction and return uncertainty (1981)
  22. The dynamics of dealer markets under competition (1983)
  23. Inferring the components of the bid ask spread (1989)
  24. Bid, ask and transaction prices in a specialist market with heterogeneously informed traders (1985)
  25. Optimal portfolio liquidation with execution cost and risk (2010)
  26. Interest rates and the credit crunch: new formulas and market models (2009)
  27. Incorporating signals into optimal trading (2019)
  28. Interest-rate modelling in collateralized markets: multiple curves, credit-liquidity effects, CCPs (2013) — link
  29. Moving from IBORs to Alternative Risk Free Rates (2020)
  30. Almost-sure hedging with permanent price impact (2016)
  31. Pricing and hedging contingent claims with liquidity costs and market impact (2013)
  32. Optimal trade execution under displaced diffusions dynamics across different risk criteria (2014)
  33. Optimal trade execution: a mean quadratic variation approach (2012)
  34. Comparison between the mean-variance optimal and the mean-quadratic-variation optimal trading strategies (2013)
  35. General intensity shapes in optimal liquidation (2015)
  36. Incorporating order-flow into optimal execution (2016)
  37. Trading algorithms with learning in latent alpha models (2019)
  38. Reinforcement learning for optimized trade execution (2006)
  39. A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
  40. Q-learning (1992)
  41. Playing atari with deep reinforcement learning (2013) — link
  42. Deep reinforcement learning with double q-learning (2016)
  43. Double deep q-learning for optimal execution (2021)
  44. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019)
  45. Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks (2019)
  46. Reinforcement learning for quantitative trading (2023)
  47. Recent advances in reinforcement learning in finance (2021) — link
  48. No-dynamic-arbitrage and market impact (2010)
  49. Price manipulation and quasi-arbitrage (2004)
  50. Adaptive market making via online learning (2013)
  51. Deep reinforcement learning for market making (2020)
  52. Model-based deep reinforcement learning for dynamic portfolio optimization (2019) — link
  53. Machine learning for market microstructure and high frequency trading (2013)
  54. Model-based reinforcement learning for predictions and control for limit order books (2019) — link
  55. Deep reinforcement learning for active high frequency trading (2021) — link
  56. Multi-agent reinforcement learning in a realistic limit order book market simulation (2020)
  57. Deep reinforcement learning for the optimal placement of cryptocurrency limit orders (2022)
  58. Continuous auctions and insider trading (1985)
  59. Deep differentiable reinforcement learning and optimal trading (2022)
  60. Optimal trading strategy and supply/demand dynamics (2013)
  61. Optimal trading with signals and stochastic price impact (2022)
  62. Liquidity fluctuations and the latent dynamics of price impact (2022)
  63. Optimal control of execution costs (1998)
  64. Direct estimation of equity market impact (2005)
  65. Fluctuations and response in financial markets: the subtle nature ofrandom’price changes (2003)
  66. How markets slowly digest changes in supply and demand (2009)
  67. Optimal basket liquidation for CARA investors is deterministic (2010)
  68. Algorithmic and high-frequency trading (2015)
  69. Reinforcement learning: an introduction MIT Press (1998)
  70. Optimal liquidation under stochastic price impact (2019)
  71. Measuring price impact and information content of trades in a time-varying setting — → 2212.12687

1 of 79 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.