Paper: arXiv 2509.12456
Authors: Rafael Zimmer, Oswaldo Luiz do Valle Costa
Abstract
Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent’s performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 6.0/10 (capped from 7.0: no out-of-sample test; see penalty flags)
- Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 7.0)
Why this score: The paper employs advanced stochastic control and Markov Decision Process formalism (high math) and implements a practical PPO agent in a custom simulator with financial risk metrics (high rigor).
Research Flowchart
flowchart TD A["Research Goal:<br>Adaptive Market Making<br>via RL in Non-Stationary LOBs"] --> B["Methodology:<br>Proximal-Policy Optimization (PPO) Agent"] B --> C["Simulator Environment<br>Inputs: Stylized Facts<br>(Clustered arrivals, Volatility, Drifts)"] C --> D["Computational Process:<br>Agent-Environment Interaction Loop"] D --> E["Comparative Evaluation:<br>RL Agent vs. Closed-Form Optimal Solution"] E --> F["Key Findings:<br>Effective Non-Stationary Adaptation<br>Validated Simulator for Pre-training"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper implements a market‑making agent using Proximal‑Policy Optimization and evaluates it in a non‑stationary limit order book simulator.
Source sentence
Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent’s performance under varying market conditions via a simulator-based environment.
Assumes 2
- The benchmark closed‑form optimal spread assumes normally distributed non‑mean reverting spreads, constant executed order size and exponentially distributed event time dynamics. model
Source sentence
The closed-form expression is derived from a simple market model which assumes normally distributed non-mean reverting spreads, constant executed order size and exponentially distributed event time dynamics.
- The training assumes no knowledge of the underlying dynamics of the environment (model‑free learning). model
Source sentence
since we assume no knowledge of the underlying dynamics of the environment when training the agent, a model-free approach is necessary.
Data
- Limit Order Book simulator with multiple non‑stationary dynamics synthetic unknown
Source sentence
Our main contribution is implementing multiple non-stationary dynamics into a single limit order book simulator, and how using fine‑controlled non‑stationary environments can enhance the resulting agent’s performance under adverse
Costs
Unclear
Finds 3
- The RL agent achieved a mean financial return of 5.203 × 10⁻⁵ (annualized ≈ +1.31 %), outperforming the benchmark’s 3.037 × 10⁻⁵ (≈ +0.76 %). performance
Source sentence
the reinforcement learning agent exhibited a mean financial return of 5.203 × 10−5 (annualized return of about +1.31%).
- The RL agent’s Sortino ratio was 0.7497, higher than the benchmark agents (0.4271 for Stoikov and –0.0079 for Long‑Only). performance
Source sentence
benchmark agent scores of 0.4271 and −0.0079 for the Stoikov and Long‑Only strategies, respectively.
- Processing latency was low: actor network 2.90 ms per episode and critic network 0.21 × 10⁻⁴ s per episode. performance
Source sentence
the actor network taking on average 2.90ms per episode, and the critic network taking 0.21 × 10−4 seconds per episode, well within the acceptable range.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (84)
- ZeroSwap: Data-driven Optimal Market Making in DeFi (2023) — link
- IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making — → 2308.08918
- Over-the-Counter Market Making via Reinforcement Learning — → 2307.01816
- Market Making of Options via Reinforcement Learning — → 2307.01814
- Integrating Tick-level Data and Periodical Signal for High-frequency Market Making — → 2306.17179
- Systematic Review on Reinforcement Learning in the Field of Fintech — → 2305.07466
- Predictive Crypto-Asset Automated Market Making Architecture for Decentralized Finance using Deep Reinforcement Learning (2022) — link
- Model-based gym environments for limit order book trading (2022) — link
and 76 more
- Market Making via Reinforcement Learning in China Commodity Market (2022) — link
- Improved Method of Stock Trading under Reinforcement Learning Based on DRQN and Sentiment Indicators ARBR (2021) — link
- Using Echo State Networks to Approximate Value Functions for Control (2021) — link
- Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making (2020) — link
- Deep Reinforcement Learning in Cryptocurrency Market Making (2019) — link
- Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019) — link
- Optimizing Market Making using Multi-Agent Reinforcement Learning (2018) — link
- Closed-form approximations in multi-asset market making (2018) — link
- Market Making via Reinforcement Learning (2018) — link
- Can maker-taker fees prevent algorithmic cooperation in market making? (2022) — link
- Effect of Market Spread Over Reinforcement Learning Based Market Maker (2019) — link
- Reinforcement Learning Approaches to Optimal Market Making (2021) — link
- Market Making With Signals Through Deep Reinforcement Learning (2021) — link
- Automated market maker inventory management with deep reinforcement learning (2023) — link
- Market Making Strategy Optimization via Deep Reinforcement Learning (2022) — link
- Multi-Asset Market Making via Multi-Task Deep Reinforcement Learning (2022) — link
- Market Making with Deep Reinforcement Learning from Limit Order Books — → 2305.15821
- Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model (2022) — link
- Algorithmic market making for options (2021) — link
- Robust Market Making via Adversarial Reinforcement Learning (2020) — link
- FairMM: A Fast and Frontrunning-Resistant Crypto Market-Maker (2022) — link
- Optimal Market Making with Persistent Order Flow (2021) — link
- Market Making and Incentives Design in the Presence of a Dark Pool: A Stackelberg Actor-Critic Approach (2022) — link
- Dealer markets: A reinforcement learning mean field game approach (2023) — link
- Profit equitably: An investigation of market maker’s impact on equitable outcomes (2021) — link
- Optimal Trading on a Dynamic Curve Automated Market Maker (2022) — link
- Interactions of Market Making Algorithms: a Study on Perceived Collusion (2021) — link
- A reinforcement learning approach to improve the performance of the Avellaneda-Stoikov market-making algorithm (2022) — link
- Computational Methods for Market Making Algorithms (2022) — link
- Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2023) — link
- Forex market forecasting using machine learning: Systematic Literature Review and meta-analysis (2023) — link
- Recent advances in reinforcement learning in finance (2023) — link
- Algorithmic market making in dealer markets with hedging and market impact (2023) — link
- Optimal Liquidation, Acquisition and Market Making Problems in HFT under Hawkes Models for LOB (2022) — link
- Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market (2021) — link
- Size matters for OTC market makers: General results and dimensionality reduction techniques (2021) — link
- Cooperation between independent market makers (2022) — link
- Empirical Analysis on Market-Making Hedging Problem based on Quadratic Programming with Time-Decay and Sparse-Term (2022)
- AI-driven liquidity provision in OTC financial markets (2022) — link
- STBM: Stochastic Trading Behavior Model for Financial Markets (2021) — link
- Towards a fully RL-based Market Simulator (2021) — link
- Algorithmic trading in a microstructural limit order book model (2020) — link
- Investigating the role of utilitarian and hedonic goals in characterizing customer loyalty in E-marketplaces (2023) — link
- Approximation algorithms for the selling with preference (2020) — link
- A CLOSED-FORM SOLUTION FOR OPTIMAL ORNSTEIN-UHLENBECK DRIVEN TRADING STRATEGIES (2020) — link
- Beating the market with a bad predictive model (2023) — link
- A logarithmic market scoring rule agent-based model to evaluate prediction markets (2023) — link
- Novel Deep Learning Model with CNN and Bi-Directional LSTM for Improved Stock Market Index Prediction (2019) — link
- Predicting Stock Price Changes Based on the Limit Order Book: A Survey (2022) — link
- Temporal Difference Learning and TD-Gammon (1995) — link
- Reinforcement Learning: A Survey (1996) — link
- Reinforcement Learning: An Introduction (2018) — link
- Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
- Optimal market making (2017) — link
- High-frequency trading in a limit order book (2008) — link
- Optimal Market Making by Reinforcement Learning (2021) — link
- Dealing with the inventory risk: a solution to the market making problem (2012) — link
- Market-making with reinforcement-learning (SAC) (2020) — link
- ABIDES: Towards High-Fidelity Multi-Agent Market Simulation (2020) — link
- Simulation and estimation of an agent-based market-model with a matching engine (2021) — link
- Dealing with the inventory risk: a solution to the market making problem (2012) — link
- Algorithmic and High-Frequency Trading (2015) — link
- Market Microstructure Theory (1995)
- Bid, Ask, and Transaction Prices in a Specialist Market with Heterogeneously Informed Traders (1985)
- Aspects of the Theory of Risk-Bearing (1965)
- Risk Aversion in the Small and in the Large (1964) — link
- Proximal Policy Optimization Algorithms (2017) — link
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018) — link
- High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015) — link
- Network Structure and Systemic Risk in Banking Systems (2010) — link
- Trades, Quotes and Prices: Financial Markets Under the Microscope (2018)
- JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading — → 2308.13289
- Human-level control through deep reinforcement learning (2015) — link
- Deep Reinforcement Learning with Double Q-learning (2015) — link
- On the Theory of the Brownian Motion (1930) — link
- A Theory of the Term Structure of Interest Rates (1985) — link
7 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.