Paper: arXiv 2410.14504
Authors: Luca Lalor, Anatoliy Swishchuk
Abstract
We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics. We begin by discussing the basics of RL and the deep RL framework used, where we deployed the state-of-the-art Soft Actor-Critic (SAC) algorithm for the deep learning part. The SAC algorithm is an off-policy entropy maximization algorithm more suitable for tackling complex, high-dimensional problems with continuous state and action spaces like in optimal market-making (MM). We introduce the optimal MM problem considered, where we detail all the deterministic and stochastic processes that go into setting up an environment for simulating this strategy. Here we also give an in-depth overview of the jump-diffusion pricing dynamics used, our method for dealing with adverse selection within the limit order book, and we highlight the working parts of our optimization problem. Next, we discuss training and testing results, where we give visuals of how important deterministic and stochastic processes such as the bid/ask, trade executions, inventory, and the reward function evolved. We include a discussion on the limitations of these results, which are important points to note for most diffusion models in this setting.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 4.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper employs advanced stochastic calculus and deep reinforcement learning theory, including complex non-Markovian processes like Hawkes jump-diffusions, which drives a high math score. However, the empirical section primarily discusses simulated training and testing results with noted limitations, lacking the backtest-ready, real-market implementation details or statistical metrics typical of high-rigor work.
Research Flowchart
flowchart TD A["Research Goal:<br>Optimal Market-Making<br>in Non-Markov Dynamics"] --> B["Methodology:<br>Deep RL using<br>Soft Actor-Critic SAC"] B --> C["Input Data:<br>Hawkes Jump-Diffusion<br>Limit Order Book"] C --> D["Computational Process:<br>Training SAC on<br>Stochastic Environment"] D --> E["Key Findings:<br>Optimal Inventory &<br>Spread Management<br>Limitations Identified"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a deep reinforcement learning framework for optimal market making that uses the Soft Actor‑Critic (SAC) algorithm and focuses on price processes with semi‑Markov and Hawkes jump‑diffusion dynamics.
Source sentence
We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics.
Assumes 6
- The midprice is simulated at each time step and a constant one‑tick spread is assumed. model
Source sentence
We simulated the midprice at each time step and then assumed a constant 1 tick spread.
- Price and order arrivals are assumed to be independent. model
Source sentence
This might not be too far off reality in some financial markets and some trading regimes (volatility dependent), but often this is not the case and assuming a constant spread could in fact be highly unrealistic in certain markets. • In these types of frameworks, price and order arrivals are assumed to be independent.
- The training and testing environments are identical, ensuring unbiased machine‑learning results. data
Source sentence
The environment we used under our training and testing sets are the same, which is based on our analysis in section 3, and this ensures unbiased results in machine learning terms.
- Adverse fills are included in the simulation to avoid phantom gains. execution
Source sentence
In order to tackle this problem, we made sure to include adverse fills in the simulation process as in Lalor and Swishchuk (2024b).
Show 2 more
- Orders are assumed to be automatically at the front of the queue in the limit order book. market
Source sentence
The models in this analysis and the majority of the HFT literature, however, often just assume that orders are automatically at the front of the queue in the LOB.
- The agent’s action space is continuous, avoiding the curse of dimensionality associated with discrete actions. model
Source sentence
However, the most noticeable limita-tion is the agents discrete action space closely related to Bellmans “curse of dimensionality”.
Data
- Synthetic market‑making environment synthetic unknown
Source sentence
In the training phase, the agent interacts with the trading environment by taking actions based on the policy.
Costs
Unclear
Finds 5
- The agent starts with a negative cumulative reward at the beginning of training. negative
Source sentence
Unsurprisingly, this agent has a negative cumulative reward at the beginning of the training process as it has not been able to figure out the optimal actions to take yet.
- Cumulative reward improves steadily as training progresses. performance
Source sentence
As we get through more episodes, this slowly improves.
- When adverse fills are excluded, most terminal rewards are positive, though some extreme negative outliers remain. performance
Source sentence
It is quite apparent that the results are mostly positive when adverse fills are excluded, with some heavy left tail results with extreme negative rewards.
- When adverse fills are included, terminal rewards remain largely negative. negative
Source sentence
When adverse fills are included, these results are still quite negative.
- Testing results show significantly improved cumulative rewards compared to training, with fewer negative tail events. performance
Source sentence
It is quiet obvious that the results have significantly improved in comparison to their respective training counterparts (in terms of the exclusion/inclusion of adverse fills), in particular regarding the left tail events present in the training phase, and the histogram shows that the cumulative rewards obtained are much more positive or less negative overall in each episode.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 14 related papers). Coverage is partial, so this is neither replication nor refutation.
4 papers in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (77)
- Optimal execution of portfolio transactions (2001)
- A discussion paper for possible approaches to building a statistically valid backtesting framework (2024)
- Deep attentive survival analysis in limit order books: Estimating fill probabilities with convolutional-transformers — → 2306.05479
- Deep reinforcement learning: A brief survey (2017)
- High-frequency trading in a limit order book (2008)
- Hawkes processes in finance (2015)
- Market making and incentives design in the presence of a dark pool: a Stackelberg actor–critic approach (2023)
- The exploration-exploitation dilemma: a multidisciplinary framework (2014)
and 69 more
- Optimal control of execution costs (1998)
- Optimal control of trading algorithms: a general impulse control approach (2011)
- Price discovery without trading: Evidence from limit orders (2019)
- Optimal execution of limit and market orders with trade director, speed limiter, and fill uncertainty (2017)
- Enhancing trading strategies with order book signals (2018)
- Algorithmic and high-frequency trading (2015)
- Buy low, sell high: A high frequency trading perspective (2014)
- Algorithmic trading, stochastic control, and mutually exciting processes (2018)
- Deep reinforcement learning for algorithmic trading (2021)
- An electronic market-maker (2001)
- The price impact of order book events (2014)
- Price dynamics in a Markovian limit order market (2013)
- Portfolio selection with transaction costs (1990)
- Limit Order Book ( LOB ) shape modeling in presence of heterogeneously informed market participants (2019)
- The negative drift of a limit order fill — → 2407.16527
- High-Frequency Trading (2016) — link
- Flow toxicity and liquidity in a high-frequency world (2012)
- Semi- Markov model for market microstructure (2015)
- High frequency trading and asymptotics for small risk aversion in a Markov renewal model (2015)
- Taming the noise in reinforcement learning via soft updates (2015) — link
- Reinforcement learning approaches to optimal market making (2021)
- Deep reinforcement learning for market making under a Hawkes process-based limit order book model (2022)
- Limit order books (2013)
- Optimal market making (2017)
- Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019)
- Spectra of some self-exciting and mutually exciting point processes (1971)
- Reinforcement learning with deep energy-based policies (2017)
- Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
- A cluster process representation of a self-exciting process (1974)
- Quantitative and comparative analyses of limit order books with general compound Hawkes processes (2019)
- A dynamic limit order market with fast and slow traders (2014)
- Market Making at the Touch with Short-Term Alpha (Chap 10.4.2) (2019)
- Limit Order Book Simulations: A Review — → 2402.17359
- Algorithmic and High-Frequency Trading Problems for Semi- Markov and Hawkes Jump-Diffusion Models — → 2409.12776
- Market Simulation under Adverse Selection — → 2409.12721
- Market making under a weakly consistent limit order book model (2019)
- Deep robust reinforcement learning for practical algorithmic trading (2019)
- Fill Probabilities in a Limit Order Book with State-Dependent Stochastic Order Flows — → 2403.02572
- A deep learning approach to estimating fill probabilities in a limit order book (2022)
- Forecasting jump arrivals in stock prices: new attention-based network architecture using limit order book data (2019)
- Lifetime portfolio selection under uncertainty: The continuous-time case (1969)
- Human-level control through deep reinforcement learning (2015)
- Using full Limit Order Book for price jump prediction (2020)
- Optimal consumption and portfolio with both fixed and proportional transaction costs (2002)
- Continuous-time stochastic control and optimization with financial applications (2009)
- Deep reinforcement learning in quantitative algorithmic trading: A review (2021) — link
- Sequence level training with recurrent neural networks (2015) — link
- On stochastic optimal control and reinforcement learning by approximate inference (2013)
- Optimal Liquidation, Acquisition and Market Making Problems in HFT under Hawkes Models for LOB (2022)
- Stochastic Optimal Control Problems in LOB for Various Stochastic Models (2023)
- Deep reinforcement learning in cryptocurrency market making (2019) — link
- Extending deep reinforcement learning frameworks in cryptocurrency market making (2020) — link
- LOB modeling using Hawkes processes with a state-dependent factor (2021) — link
- Mastering the game of Go with deep neural networks and tree search (2016)
- How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries–A review and research agenda (2022)
- General Compound Hawkes Processes for Mid-Price Prediction (2021) — link
- 79+ amazing algorithmic trading statistics (2016) — link
- Stable Baselines3 (2020) — link
- A semi- Markovian modeling of limit order markets (2017)
- General semi- Markov model for limit order books (2017)
- General compound Hawkes processes in limit order books (2020)
- Compound Hawkes processes in limit order books (2019)
- Financial modelling with jump processes (2003)
- An application of deep reinforcement learning to algorithmic trading (2021)
- Robot trajectory optimization using approximate inference (2009)
- Optimal investment-consumption models with constraints (1989)
- Statistical Analysis of the Price Jumps of Financial Assets Based on LOB Data (2017)
- Deep reinforcement learning for trading (2019) — link
- Maximum entropy inverse reinforcement learning. (2008)
6 of 77 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.