Paper: arXiv 2509.12456

Authors: Rafael Zimmer, Oswaldo Luiz do Valle Costa

Abstract

Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent’s performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: no out-of-sample test; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 7.0)

Why this score: The paper employs advanced stochastic control and Markov Decision Process formalism (high math) and implements a practical PPO agent in a custom simulator with financial risk metrics (high rigor).

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Adaptive Market Making<br>via RL in Non-Stationary LOBs"] --> B["Methodology:<br>Proximal-Policy Optimization (PPO) Agent"]
  B --> C["Simulator Environment<br>Inputs: Stylized Facts<br>(Clustered arrivals, Volatility, Drifts)"]
  C --> D["Computational Process:<br>Agent-Environment Interaction Loop"]
  D --> E["Comparative Evaluation:<br>RL Agent vs. Closed-Form Optimal Solution"]
  E --> F["Key Findings:<br>Effective Non-Stationary Adaptation<br>Validated Simulator for Pre-training"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper implements a market‑making agent using Proximal‑Policy Optimization and evaluates it in a non‑stationary limit order book simulator.

Source sentence
Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent’s performance under varying market conditions via a simulator-based environment.

Assumes 2

  • The benchmark closed‑form optimal spread assumes normally distributed non‑mean reverting spreads, constant executed order size and exponentially distributed event time dynamics. model
    Source sentence
    The closed-form expression is derived from a simple market model which assumes normally distributed non-mean reverting spreads, constant executed order size and exponentially distributed event time dynamics.
  • The training assumes no knowledge of the underlying dynamics of the environment (model‑free learning). model
    Source sentence
    since we assume no knowledge of the underlying dynamics of the environment when training the agent, a model-free approach is necessary.

Data

  • Limit Order Book simulator with multiple non‑stationary dynamics synthetic unknown
    Source sentence
    Our main contribution is implementing multiple non-stationary dynamics into a single limit order book simulator, and how using fine‑controlled non‑stationary environments can enhance the resulting agent’s performance under adverse

Costs

Unclear The classifier was not confident.

Finds 3

  • The RL agent achieved a mean financial return of 5.203 × 10⁻⁵ (annualized ≈ +1.31 %), outperforming the benchmark’s 3.037 × 10⁻⁵ (≈ +0.76 %). performance
    Source sentence
    the reinforcement learning agent exhibited a mean financial return of 5.203 × 10−5 (annualized return of about +1.31%).
  • The RL agent’s Sortino ratio was 0.7497, higher than the benchmark agents (0.4271 for Stoikov and –0.0079 for Long‑Only). performance
    Source sentence
    benchmark agent scores of 0.4271 and −0.0079 for the Stoikov and Long‑Only strategies, respectively.
  • Processing latency was low: actor network 2.90 ms per episode and critic network 0.21 × 10⁻⁴ s per episode. performance
    Source sentence
    the actor network taking on average 2.90ms per episode, and the critic network taking 0.21 × 10−4 seconds per episode, well within the acceptable range.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (84)

  1. ZeroSwap: Data-driven Optimal Market Making in DeFi (2023) — link
  2. IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making — → 2308.08918
  3. Over-the-Counter Market Making via Reinforcement Learning — → 2307.01816
  4. Market Making of Options via Reinforcement Learning — → 2307.01814
  5. Integrating Tick-level Data and Periodical Signal for High-frequency Market Making — → 2306.17179
  6. Systematic Review on Reinforcement Learning in the Field of Fintech — → 2305.07466
  7. Predictive Crypto-Asset Automated Market Making Architecture for Decentralized Finance using Deep Reinforcement Learning (2022) — link
  8. Model-based gym environments for limit order book trading (2022) — link
and 76 more
  1. Market Making via Reinforcement Learning in China Commodity Market (2022) — link
  2. Improved Method of Stock Trading under Reinforcement Learning Based on DRQN and Sentiment Indicators ARBR (2021) — link
  3. Using Echo State Networks to Approximate Value Functions for Control (2021) — link
  4. Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making (2020) — link
  5. Deep Reinforcement Learning in Cryptocurrency Market Making (2019) — link
  6. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019) — link
  7. Optimizing Market Making using Multi-Agent Reinforcement Learning (2018) — link
  8. Closed-form approximations in multi-asset market making (2018) — link
  9. Market Making via Reinforcement Learning (2018) — link
  10. Can maker-taker fees prevent algorithmic cooperation in market making? (2022) — link
  11. Effect of Market Spread Over Reinforcement Learning Based Market Maker (2019) — link
  12. Reinforcement Learning Approaches to Optimal Market Making (2021) — link
  13. Market Making With Signals Through Deep Reinforcement Learning (2021) — link
  14. Automated market maker inventory management with deep reinforcement learning (2023) — link
  15. Market Making Strategy Optimization via Deep Reinforcement Learning (2022) — link
  16. Multi-Asset Market Making via Multi-Task Deep Reinforcement Learning (2022) — link
  17. Market Making with Deep Reinforcement Learning from Limit Order Books — → 2305.15821
  18. Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model (2022) — link
  19. Algorithmic market making for options (2021) — link
  20. Robust Market Making via Adversarial Reinforcement Learning (2020) — link
  21. FairMM: A Fast and Frontrunning-Resistant Crypto Market-Maker (2022) — link
  22. Optimal Market Making with Persistent Order Flow (2021) — link
  23. Market Making and Incentives Design in the Presence of a Dark Pool: A Stackelberg Actor-Critic Approach (2022) — link
  24. Dealer markets: A reinforcement learning mean field game approach (2023) — link
  25. Profit equitably: An investigation of market maker’s impact on equitable outcomes (2021) — link
  26. Optimal Trading on a Dynamic Curve Automated Market Maker (2022) — link
  27. Interactions of Market Making Algorithms: a Study on Perceived Collusion (2021) — link
  28. A reinforcement learning approach to improve the performance of the Avellaneda-Stoikov market-making algorithm (2022) — link
  29. Computational Methods for Market Making Algorithms (2022) — link
  30. Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2023) — link
  31. Forex market forecasting using machine learning: Systematic Literature Review and meta-analysis (2023) — link
  32. Recent advances in reinforcement learning in finance (2023) — link
  33. Algorithmic market making in dealer markets with hedging and market impact (2023) — link
  34. Optimal Liquidation, Acquisition and Market Making Problems in HFT under Hawkes Models for LOB (2022) — link
  35. Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market (2021) — link
  36. Size matters for OTC market makers: General results and dimensionality reduction techniques (2021) — link
  37. Cooperation between independent market makers (2022) — link
  38. Empirical Analysis on Market-Making Hedging Problem based on Quadratic Programming with Time-Decay and Sparse-Term (2022)
  39. AI-driven liquidity provision in OTC financial markets (2022) — link
  40. STBM: Stochastic Trading Behavior Model for Financial Markets (2021) — link
  41. Towards a fully RL-based Market Simulator (2021) — link
  42. Algorithmic trading in a microstructural limit order book model (2020) — link
  43. Investigating the role of utilitarian and hedonic goals in characterizing customer loyalty in E-marketplaces (2023) — link
  44. Approximation algorithms for the selling with preference (2020) — link
  45. A CLOSED-FORM SOLUTION FOR OPTIMAL ORNSTEIN-UHLENBECK DRIVEN TRADING STRATEGIES (2020) — link
  46. Beating the market with a bad predictive model (2023) — link
  47. A logarithmic market scoring rule agent-based model to evaluate prediction markets (2023) — link
  48. Novel Deep Learning Model with CNN and Bi-Directional LSTM for Improved Stock Market Index Prediction (2019) — link
  49. Predicting Stock Price Changes Based on the Limit Order Book: A Survey (2022) — link
  50. Temporal Difference Learning and TD-Gammon (1995) — link
  51. Reinforcement Learning: A Survey (1996) — link
  52. Reinforcement Learning: An Introduction (2018) — link
  53. Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
  54. Optimal market making (2017) — link
  55. High-frequency trading in a limit order book (2008) — link
  56. Optimal Market Making by Reinforcement Learning (2021) — link
  57. Dealing with the inventory risk: a solution to the market making problem (2012) — link
  58. Market-making with reinforcement-learning (SAC) (2020) — link
  59. ABIDES: Towards High-Fidelity Multi-Agent Market Simulation (2020) — link
  60. Simulation and estimation of an agent-based market-model with a matching engine (2021) — link
  61. Dealing with the inventory risk: a solution to the market making problem (2012) — link
  62. Algorithmic and High-Frequency Trading (2015) — link
  63. Market Microstructure Theory (1995)
  64. Bid, Ask, and Transaction Prices in a Specialist Market with Heterogeneously Informed Traders (1985)
  65. Aspects of the Theory of Risk-Bearing (1965)
  66. Risk Aversion in the Small and in the Large (1964) — link
  67. Proximal Policy Optimization Algorithms (2017) — link
  68. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018) — link
  69. High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015) — link
  70. Network Structure and Systemic Risk in Banking Systems (2010) — link
  71. Trades, Quotes and Prices: Financial Markets Under the Microscope (2018)
  72. JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading — → 2308.13289
  73. Human-level control through deep reinforcement learning (2015) — link
  74. Deep Reinforcement Learning with Double Q-learning (2015) — link
  75. On the Theory of the Brownian Motion (1930) — link
  76. A Theory of the Term Structure of Interest Rates (1985) — link

7 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.