Paper: arXiv 2609.11614

Authors: Felipe Moret, Fabrizio Lillo

Abstract

Classical market-making strategies based on stochastic control, such as the Avellaneda-Stoikov and the Guéant-Lehalle-Fernandez-Tapia (GLFT) extension, provide closed-form quoting rules, but rest on assumptions that break down at realistic microstructure timescales. One of them is that order flow is stationary, while empirical evidence points to the existence of regimes, possibly associated with algorithmic execution of metaorders. In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM) - a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book. We find that, in the stationary setting, RLMM outperforms GLFT across the entire observed risk-return frontier. The RLMM is more robust to flow asymmetry than GLFT, but, like any stationarily trained strategy, it still suffers large drawdowns from inventory saturation under persistent directional imbalance. Augmenting the state of RLMM with two auxiliary signals - a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance -restores profitability. A final scenario-bandit step that reweights low-return regime scenarios further improves performance under random-persistence and correlated-direction stress.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: no out-of-sample test; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a novel deep reinforcement learning approach to market making, addressing the critical issue of regime-switching order flow. It demonstrates strong empirical rigor through its use of a queue-aware event-driven simulator and robust testing against established benchmarks. The methodology is clearly articulated, and the provision of code enhances reproducibility.

Research Flowchart

  flowchart TD
    A[Research Goal: Robust Market Making under Regime-Switching Order Flow] --> B{Methodology: Deep Reinforcement Learning};
    B --> C[Data/Inputs: Zero-Intelligence Limit Order Book];
    C --> D{Computational Process: Rainbow-style Distributional DQN (C51)};
    D --> E[Outcome 1: RLMM outperforms GLFT in stationary setting];
    D --> F[Outcome 2: Robustness to flow asymmetry with auxiliary signals];
    D --> G[Outcome 3: Further improved performance with scenario-bandit reweighting];

Cited References (40)

  1. Bayesian Online Changepoint Detection (2007) — link
  2. High-frequency trading in a limit order book (2008) — link
  3. A Distributional Perspective on Reinforcement Learning (2017) — link
  4. How Markets Slowly Digest Changes in Supply and Demand (2009)
  5. Trades, Quotes and Prices: Financial Markets under the Microscope (2018) — link
  6. Buy Low, Sell High: A High Frequency Trading Perspective (2014) — link
  7. Algorithmic and High-Frequency Trading (2015)
  8. The Predictive Power of Zero Intelligence in Financial Markets (2005) — link
and 32 more
  1. On-line Inference for Multiple Changepoint Problems (2007) — link
  2. Noisy Networks for Exploration (2018) — link
  3. Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
  4. Reinforcement Learning Approaches to Optimal Market Making (2021) — link
  5. Market Making with Signals Through Deep Reinforcement Learning (2021) — link
  6. Dealing with the Inventory Risk: A Solution to the Market Making Problem (2013) — link
  7. General Intensity Shapes in Optimal Liquidation (2015) — link
  8. The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making (2016)
  9. Optimal Market Making (2017) — link
  10. Deep Reinforcement Learning for Market Making in Corporate Bonds: Beating the Curse of Dimensionality (2019) — link
  11. A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle (1989) — link
  12. Rainbow: Combining Improvements in Deep Reinforcement Learning (2018) — link
  13. Modern Perspectives on Reinforcement Learning in Finance (2020) — link
  14. Optimal Posting Price of Limit Orders: Learning by Trading (2013) — link
  15. Market Microstructure in Practice (2018) — link
  16. The Long Memory of the Efficient Market (2004) — link
  17. Human-Level Control Through Deep Reinforcement Learning (2015) — link
  18. Markov Decision Processes: Discrete Stochastic Dynamic Programming (1994)
  19. Deep Reinforcement Learning in Cryptocurrency Market Making (2019) — link
  20. Prioritized Experience Replay (2016) — link
  21. Statistical Theory of the Continuous Double Auction (2003) — link
  22. Market Making via Reinforcement Learning (2018) — link
  23. Robust Market Making via Adversarial Reinforcement Learning (2020) — link
  24. Learning to Predict by the Methods of Temporal Differences (1988) — link
  25. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning (1999) — link
  26. Reinforcement Learning: An Introduction (2018)
  27. Why is Equity Order Flow So Persistent? (2015) — link
  28. Online Learning of Order Flow and Market Impact with Bayesian Change-Point Detection Methods — → 2307.02375
  29. Deep Reinforcement Learning with Double Q -Learning (2016) — link
  30. Dueling Network Architectures for Deep Reinforcement Learning (2016) — link
  31. Q -learning (1992) — link
  32. Deep Reinforcement Learning for Trading (2020) — link

1 of 40 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.