Paper: arXiv 2410.14504

Authors: Luca Lalor, Anatoliy Swishchuk

Abstract

We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics. We begin by discussing the basics of RL and the deep RL framework used, where we deployed the state-of-the-art Soft Actor-Critic (SAC) algorithm for the deep learning part. The SAC algorithm is an off-policy entropy maximization algorithm more suitable for tackling complex, high-dimensional problems with continuous state and action spaces like in optimal market-making (MM). We introduce the optimal MM problem considered, where we detail all the deterministic and stochastic processes that go into setting up an environment for simulating this strategy. Here we also give an in-depth overview of the jump-diffusion pricing dynamics used, our method for dealing with adverse selection within the limit order book, and we highlight the working parts of our optimization problem. Next, we discuss training and testing results, where we give visuals of how important deterministic and stochastic processes such as the bid/ask, trade executions, inventory, and the reward function evolved. We include a discussion on the limitations of these results, which are important points to note for most diffusion models in this setting.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper employs advanced stochastic calculus and deep reinforcement learning theory, including complex non-Markovian processes like Hawkes jump-diffusions, which drives a high math score. However, the empirical section primarily discusses simulated training and testing results with noted limitations, lacking the backtest-ready, real-market implementation details or statistical metrics typical of high-rigor work.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Optimal Market-Making<br>in Non-Markov Dynamics"] --> B["Methodology:<br>Deep RL using<br>Soft Actor-Critic SAC"]
  B --> C["Input Data:<br>Hawkes Jump-Diffusion<br>Limit Order Book"]
  C --> D["Computational Process:<br>Training SAC on<br>Stochastic Environment"]
  D --> E["Key Findings:<br>Optimal Inventory &<br>Spread Management<br>Limitations Identified"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a deep reinforcement learning framework for optimal market making that uses the Soft Actor‑Critic (SAC) algorithm and focuses on price processes with semi‑Markov and Hawkes jump‑diffusion dynamics.

Source sentence
We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics.

Assumes 6

  • The midprice is simulated at each time step and a constant one‑tick spread is assumed. model
    Source sentence
    We simulated the midprice at each time step and then assumed a constant 1 tick spread.
  • Price and order arrivals are assumed to be independent. model
    Source sentence
    This might not be too far off reality in some financial markets and some trading regimes (volatility dependent), but often this is not the case and assuming a constant spread could in fact be highly unrealistic in certain markets. • In these types of frameworks, price and order arrivals are assumed to be independent.
  • The training and testing environments are identical, ensuring unbiased machine‑learning results. data
    Source sentence
    The environment we used under our training and testing sets are the same, which is based on our analysis in section 3, and this ensures unbiased results in machine learning terms.
  • Adverse fills are included in the simulation to avoid phantom gains. execution
    Source sentence
    In order to tackle this problem, we made sure to include adverse fills in the simulation process as in Lalor and Swishchuk (2024b).
Show 2 more
  • Orders are assumed to be automatically at the front of the queue in the limit order book. market
    Source sentence
    The models in this analysis and the majority of the HFT literature, however, often just assume that orders are automatically at the front of the queue in the LOB.
  • The agent’s action space is continuous, avoiding the curse of dimensionality associated with discrete actions. model
    Source sentence
    However, the most noticeable limita-tion is the agents discrete action space closely related to Bellmans “curse of dimensionality”.

Data

  • Synthetic market‑making environment synthetic unknown
    Source sentence
    In the training phase, the agent interacts with the trading environment by taking actions based on the policy.

Costs

Unclear The classifier was not confident.

Finds 5

  • The agent starts with a negative cumulative reward at the beginning of training. negative
    Source sentence
    Unsurprisingly, this agent has a negative cumulative reward at the beginning of the training process as it has not been able to figure out the optimal actions to take yet.
  • Cumulative reward improves steadily as training progresses. performance
    Source sentence
    As we get through more episodes, this slowly improves.
  • When adverse fills are excluded, most terminal rewards are positive, though some extreme negative outliers remain. performance
    Source sentence
    It is quite apparent that the results are mostly positive when adverse fills are excluded, with some heavy left tail results with extreme negative rewards.
  • When adverse fills are included, terminal rewards remain largely negative. negative
    Source sentence
    When adverse fills are included, these results are still quite negative.
  • Testing results show significantly improved cumulative rewards compared to training, with fewer negative tail events. performance
    Source sentence
    It is quiet obvious that the results have significantly improved in comparison to their respective training counterparts (in terms of the exclusion/inclusion of adverse fills), in particular regarding the left tail events present in the training phase, and the histogram shows that the cumulative rewards obtained are much more positive or less negative overall in each episode.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 14 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 4 papers in the archive (2 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (77)

  1. Optimal execution of portfolio transactions (2001)
  2. A discussion paper for possible approaches to building a statistically valid backtesting framework (2024)
  3. Deep attentive survival analysis in limit order books: Estimating fill probabilities with convolutional-transformers — → 2306.05479
  4. Deep reinforcement learning: A brief survey (2017)
  5. High-frequency trading in a limit order book (2008)
  6. Hawkes processes in finance (2015)
  7. Market making and incentives design in the presence of a dark pool: a Stackelberg actor–critic approach (2023)
  8. The exploration-exploitation dilemma: a multidisciplinary framework (2014)
and 69 more
  1. Optimal control of execution costs (1998)
  2. Optimal control of trading algorithms: a general impulse control approach (2011)
  3. Price discovery without trading: Evidence from limit orders (2019)
  4. Optimal execution of limit and market orders with trade director, speed limiter, and fill uncertainty (2017)
  5. Enhancing trading strategies with order book signals (2018)
  6. Algorithmic and high-frequency trading (2015)
  7. Buy low, sell high: A high frequency trading perspective (2014)
  8. Algorithmic trading, stochastic control, and mutually exciting processes (2018)
  9. Deep reinforcement learning for algorithmic trading (2021)
  10. An electronic market-maker (2001)
  11. The price impact of order book events (2014)
  12. Price dynamics in a Markovian limit order market (2013)
  13. Portfolio selection with transaction costs (1990)
  14. Limit Order Book ( LOB ) shape modeling in presence of heterogeneously informed market participants (2019)
  15. The negative drift of a limit order fill — → 2407.16527
  16. High-Frequency Trading (2016) — link
  17. Flow toxicity and liquidity in a high-frequency world (2012)
  18. Semi- Markov model for market microstructure (2015)
  19. High frequency trading and asymptotics for small risk aversion in a Markov renewal model (2015)
  20. Taming the noise in reinforcement learning via soft updates (2015) — link
  21. Reinforcement learning approaches to optimal market making (2021)
  22. Deep reinforcement learning for market making under a Hawkes process-based limit order book model (2022)
  23. Limit order books (2013)
  24. Optimal market making (2017)
  25. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019)
  26. Spectra of some self-exciting and mutually exciting point processes (1971)
  27. Reinforcement learning with deep energy-based policies (2017)
  28. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  29. A cluster process representation of a self-exciting process (1974)
  30. Quantitative and comparative analyses of limit order books with general compound Hawkes processes (2019)
  31. A dynamic limit order market with fast and slow traders (2014)
  32. Market Making at the Touch with Short-Term Alpha (Chap 10.4.2) (2019)
  33. Limit Order Book Simulations: A Review — → 2402.17359
  34. Algorithmic and High-Frequency Trading Problems for Semi- Markov and Hawkes Jump-Diffusion Models — → 2409.12776
  35. Market Simulation under Adverse Selection — → 2409.12721
  36. Market making under a weakly consistent limit order book model (2019)
  37. Deep robust reinforcement learning for practical algorithmic trading (2019)
  38. Fill Probabilities in a Limit Order Book with State-Dependent Stochastic Order Flows — → 2403.02572
  39. A deep learning approach to estimating fill probabilities in a limit order book (2022)
  40. Forecasting jump arrivals in stock prices: new attention-based network architecture using limit order book data (2019)
  41. Lifetime portfolio selection under uncertainty: The continuous-time case (1969)
  42. Human-level control through deep reinforcement learning (2015)
  43. Using full Limit Order Book for price jump prediction (2020)
  44. Optimal consumption and portfolio with both fixed and proportional transaction costs (2002)
  45. Continuous-time stochastic control and optimization with financial applications (2009)
  46. Deep reinforcement learning in quantitative algorithmic trading: A review (2021) — link
  47. Sequence level training with recurrent neural networks (2015) — link
  48. On stochastic optimal control and reinforcement learning by approximate inference (2013)
  49. Optimal Liquidation, Acquisition and Market Making Problems in HFT under Hawkes Models for LOB (2022)
  50. Stochastic Optimal Control Problems in LOB for Various Stochastic Models (2023)
  51. Deep reinforcement learning in cryptocurrency market making (2019) — link
  52. Extending deep reinforcement learning frameworks in cryptocurrency market making (2020) — link
  53. LOB modeling using Hawkes processes with a state-dependent factor (2021) — link
  54. Mastering the game of Go with deep neural networks and tree search (2016)
  55. How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries–A review and research agenda (2022)
  56. General Compound Hawkes Processes for Mid-Price Prediction (2021) — link
  57. 79+ amazing algorithmic trading statistics (2016) — link
  58. Stable Baselines3 (2020) — link
  59. A semi- Markovian modeling of limit order markets (2017)
  60. General semi- Markov model for limit order books (2017)
  61. General compound Hawkes processes in limit order books (2020)
  62. Compound Hawkes processes in limit order books (2019)
  63. Financial modelling with jump processes (2003)
  64. An application of deep reinforcement learning to algorithmic trading (2021)
  65. Robot trajectory optimization using approximate inference (2009)
  66. Optimal investment-consumption models with constraints (1989)
  67. Statistical Analysis of the Price Jumps of Financial Assets Based on LOB Data (2017)
  68. Deep reinforcement learning for trading (2019) — link
  69. Maximum entropy inverse reinforcement learning. (2008)

6 of 77 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.