Paper: arXiv 2407.21025

Authors: Yuheng Zheng, Zihan Ding

Abstract

This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We bridge the modern RL theory and the continuous-time statistical models in high-frequency financial economics. Different with most existing literature on methodological research about developing various RL methods for market making problem, our work is a pilot to provide the theoretical analysis. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment \(Δ\) \(-\) as \(Δ\) becomes smaller, the error will be smaller but the complexity will be larger. We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as \(Δ\rightarrow0\). The Nash Q-learning algorithm, which is an online multi-agent RL method, is applied to solve the equilibrium. Our theories are not only useful for practitioners to choose the sampling frequency, but also very general and applicable to other high-frequency financial decision making problems, e.g., optimal executions, as long as the time-discretization of a continuous-time markov decision process is adopted. Monte Carlo simulation evidence support all of our theories.

Complexity vs Empirical Score

  • Math Complexity: 9.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper is highly theoretical with dense mathematical analysis of continuous-time models, convergence, and sample complexity, yet it includes Monte Carlo simulations to support the theories, indicating moderate empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Establish theoretical analysis for RL<br>in high-frequency market making"] --> B["Methodology: Continuous-Time &<br>Game Theory Framework"]
  B --> C["Key Analysis: Effect of Sampling Frequency Δ<br>Tradeoff: Error ↓ vs Complexity ↑ as Δ→0"]
  B --> D["Multi-Agent Extension<br>General-Sum Game & Nash Q-Learning"]
  C & D --> E["Computational Process<br>Monte Carlo Simulation"]
  E --> F["Key Outcomes & Findings"]
  F --> G["Convergence Proof<br>Nash Eq → Continuous-Time Eq as Δ→0"]
  F --> H["Practical Guidance<br>Optimal Δ selection for practitioners"]
  F --> I["General Applicability<br>Extends to optimal execution & other HF problems"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper provides a comprehensive theoretical analysis of reinforcement learning for high‑frequency market making, including convergence of discretized models to continuous‑time limits, sample‑complexity upper bounds, and a model‑free Nash Q‑learning algorithm for multi‑agent equilibrium.

Source sentence
This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger.

Assumes 5

  • The transition rate functions λ_{k,l}(a) are bounded away from zero and above by a constant C_λ > 0 for all states and actions. model
    Source sentence
    We propose the following assumption on the rate function λk,l. Assumption 1. For the mid-price, the transition rate functions λk,l(a) satisfy that, there exists a constant Cλ > 0 such that 0 < λk,l(a) < Cλ for any k, l and a ∈A. Denote by N a t (resp. N b t ) a controlled Poisson process with intensity λ(|pa t −Xt|) (resp. λ(|pb t − Xt|)), which models the coming flow of the market buy (resp. sell) order at time t.
  • The mid‑price dynamics follow a continuous‑time Markov chain with a (possibly action‑dependent) Q‑matrix, and market order flow is modeled by controlled Poisson processes. market
    Source sentence
    The mid-price Xt is a continuous-time Markov chain on state space SX with the transition 6 rate matrix (a.k.a. Q-matrix) QX(a) defined as QX(a) :=   −λ1,2(a) λ1,2(a) 0 0 0 λ2,1(a) −(λ2,1(a) + λ2,3(a)) λ2,3(a) 0 0 0 ... ... ... 0 0 0 ... ... ...
  • The optimal policies in the continuous‑time and discretized MDPs are stationary Markov policies. model
    Source sentence
    Following Kakumanu (1971) and Section 1.7 in Gihman and Skorohod (2012), for the MDP M0 (resp. M∆), if a policy can be expressed as at = π(St) (resp. ai = π(Si)) where π : S →A is a deterministic mapping, then we say this policy is a stationary Markov policy and we denote this policy as π(·). Theorem 1. Under Assumption 1, there exist stationary Markov policies π∗ 0(·) and π∗ ∆(·), such that the optimal value
  • The learning rate β(n)(s,a) follows a polynomial decay β = (N(s,a,n))^{-ω} with ω∈(1/2,1). other
    Source sentence
    Following Even-Dar et al. (2003), we use a polynomial learning rate β(n)(s, a) which is given by β(n)(s, a) = (N(s, a, n))−ω, where ω ∈(1 2, 1) and N(s, a, n) is the one plus the number of times, until the nth iteration, that we visited the state-action pair (s, a).
Show 1 more
  • The sample‑complexity analysis uses a definition based on the value‑function error rather than the Q‑function error. other
    Source sentence
    Our definition of sample complexity is slightly different with that of Even-Dar et al. (2003) in that, we use the value function error while Even-Dar et al. (2003) used the Q function error. Theorem 2.

Data

  • Monte Carlo simulation synthetic public
    Source sentence
    Monte Carlo simulation evidence support all of our theories. To capture the sample complexity using numerical simulations, we record the iteration number N∆, which is the smallest iteration steps n such that ||V (n) ∆(·) −V ∗ ∆(·)|| ≤0.1, when applying the Q-learning algorithm to the discrete-time MDP for each value of ∆.

Costs

Unclear - sources disagree The platform classifier and the paper-text extraction give different answers, so no value is asserted.

Platform classifier: Not applicable

Paper text extraction: Not modeled

Source sentence (the authors' words)
The reward functions of our models in Sections 2 and 3.1 do not include the transaction costs due to the convenience of technical analysis.

Finds 4

  • A trade‑off exists: decreasing the time increment Δ reduces learning error but increases sample‑complexity (and thus transaction‑cost exposure). theoretical
    Source sentence
    We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger. Regarding the upper bound εV + CV ∆as a metric of the learning error and the upper bound in (6) as a metric of the sample complexity, we have the tradeoffas
  • The sample‑complexity upper bound is polynomial in all model parameters and decreases with Δ, diverging as Δ→0. theoretical
    Source sentence
    A notable and desirable result in Theorem 2 is that, the sample complexity upper bound is polynomial, rather than exponential, in all model parameters. In (6), the exponents of ∆are both negative, since 6−2 ω −(|SX|+|SY |)(3+ 1 ω) ≤6−2 ω −2(3+ 1 ω) ≤−4 ω < 0 and (1−|SX|−|SY |) 1 1−ω ≤ − 1 1−ω < 0.
  • Numerical experiments confirm that the required number of Q‑learning iterations grows as Δ decreases, validating the theoretical sample‑complexity bound. performance
    Source sentence
    This increasing trend of the iteration numbers demonstrate our theory about the sample complexity, i.e., the sample complexity upper bound will increase when ∆decreases. So, the only difference in the upper bound is the values of ∆and the upward trend in Figure 3 does validate Theorem 2.
  • The Nash equilibrium of the discretized two‑player game converges to the continuous‑time Nash equilibrium as Δ→0. theoretical
    Source sentence
    We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as $Δ\rightarrow0$. A pair of strategies (π1,∗ ∆, π2,∗ ∆) ∈Π1 ∆× Π2 ∆is called a Nash equilibrium if V 1,π1,∗ ∆,π2,∗ ∆ ∆ (s) ≥V 1,π1,π2,∗ ∆ ∆ (s) and V 2,π1,∗ ∆,π2,∗ ∆ ∆ (s) ≥V 2,π1,∗ ∆,π2 ∆ (s), for any (π1, π2) ∈Π1 ∆× Π2 ∆and s ∈S. In Theorem 5 below, we

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 23 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 4 papers in the archive (4 by independent authors)

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (41)

  1. Robust Market Making: To Quote, or not To Quote — → 2508.16588
  2. Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2023) — link
  3. High frequency market making: The role of speed (2023) — link
  4. Can maker-taker fees prevent algorithmic cooperation in market making? (2022) — link
  5. AI-driven liquidity provision in OTC financial markets (2022) — link
  6. Towards multi‐agent reinforcement learning‐driven over‐the‐counter market simulations (2022) — link
  7. Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality (2022) — link
  8. Recent advances in reinforcement learning in finance (2021) — link
and 33 more
  1. Interactions of market making algorithms: a study on perceived collusion (2021) — link
  2. Reinforcement Learning Approaches to Optimal Market Making (2021) — link
  3. Towards a fully rl-based market simulator (2021) — link
  4. Dynamic Equilibrium of Market Making with Price Competition (2020) — link
  5. Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
  6. Financial Machine Learning (2019) — link
  7. Q-learning with Nearest Neighbors (2018) — link
  8. High-Frequency Financial Econometrics (2016) — link
  9. Buy Low, Sell High: A High Frequency Trading Perspective (2014) — link
  10. Dealing with the inventory risk: a solution to the market making problem (2011) — link
  11. A Stochastic Model for Order Book Dynamics (2010) — link
  12. High-frequency trading in a limit order book (2008) — link
  13. Applied stochastic processes and control for jump-diffusions - modeling, analysis, and computation (2007)
  14. Games with Imperfectly Observable Actions in Continuous Time (2007) — link
  15. Nonzero-sum games for continuous-time Markov chains with unbounded discounted payoffs (2005) — link
  16. Learning Rates for Q-learning (2004) — link
  17. Nash Q-Learning for General-Sum Stochastic Games (2003)
  18. Competitive Markov Decision Processes (1996) — link
  19. Controlled markov processes and viscosity solutions (1994) — link
  20. Towards a Theory of Discounted Repeated Games with Imperfect Monitoring (1990) — link
  21. Informed Speculation with Imperfect Competition (1989) — link
  22. Continuous Auctions and Insider Trading (1985) — link
  23. On the Convergence of the Discrete Time Dynamic Programming Equation for General Semigroups (1982) — link
  24. Optimal dealer pricing under transactions and return uncertainty (1981) — link
  25. Dealership market: Market-making with inventory (1980) — link
  26. Continuously Discounted Markov Decision Model with Countable State and Action Space (1971) — link
  27. Algorithmic Collusion in Electronic Markets: The Impact of Tick Size (2022) — link
  28. Sample Complexity Bounds of Exploration (2012) — link
  29. Reinforcement Learning: An Introduction (2005) — link
  30. Non-Cooperative Games (2003)
  31. Market structure, information, futures markets, and price formation. (1984)
  32. Controlled Stochastic Processes (1979) — link
  33. Assumption 2

1 of 41 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.