Paper: arXiv 2407.21025
Authors: Yuheng Zheng, Zihan Ding
Abstract
This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We bridge the modern RL theory and the continuous-time statistical models in high-frequency financial economics. Different with most existing literature on methodological research about developing various RL methods for market making problem, our work is a pilot to provide the theoretical analysis. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment \(Δ\) \(-\) as \(Δ\) becomes smaller, the error will be smaller but the complexity will be larger. We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as \(Δ\rightarrow0\). The Nash Q-learning algorithm, which is an online multi-agent RL method, is applied to solve the equilibrium. Our theories are not only useful for practitioners to choose the sampling frequency, but also very general and applicable to other high-frequency financial decision making problems, e.g., optimal executions, as long as the time-discretization of a continuous-time markov decision process is adopted. Monte Carlo simulation evidence support all of our theories.
Complexity vs Empirical Score
- Math Complexity: 9.0/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper is highly theoretical with dense mathematical analysis of continuous-time models, convergence, and sample complexity, yet it includes Monte Carlo simulations to support the theories, indicating moderate empirical rigor.
Research Flowchart
flowchart TD A["Research Goal<br>Establish theoretical analysis for RL<br>in high-frequency market making"] --> B["Methodology: Continuous-Time &<br>Game Theory Framework"] B --> C["Key Analysis: Effect of Sampling Frequency Δ<br>Tradeoff: Error ↓ vs Complexity ↑ as Δ→0"] B --> D["Multi-Agent Extension<br>General-Sum Game & Nash Q-Learning"] C & D --> E["Computational Process<br>Monte Carlo Simulation"] E --> F["Key Outcomes & Findings"] F --> G["Convergence Proof<br>Nash Eq → Continuous-Time Eq as Δ→0"] F --> H["Practical Guidance<br>Optimal Δ selection for practitioners"] F --> I["General Applicability<br>Extends to optimal execution & other HF problems"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper provides a comprehensive theoretical analysis of reinforcement learning for high‑frequency market making, including convergence of discretized models to continuous‑time limits, sample‑complexity upper bounds, and a model‑free Nash Q‑learning algorithm for multi‑agent equilibrium.
Source sentence
This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger.
Assumes 5
- The transition rate functions λ_{k,l}(a) are bounded away from zero and above by a constant C_λ > 0 for all states and actions. model
Source sentence
We propose the following assumption on the rate function λk,l. Assumption 1. For the mid-price, the transition rate functions λk,l(a) satisfy that, there exists a constant Cλ > 0 such that 0 < λk,l(a) < Cλ for any k, l and a ∈A. Denote by N a t (resp. N b t ) a controlled Poisson process with intensity λ(|pa t −Xt|) (resp. λ(|pb t − Xt|)), which models the coming flow of the market buy (resp. sell) order at time t.
- The mid‑price dynamics follow a continuous‑time Markov chain with a (possibly action‑dependent) Q‑matrix, and market order flow is modeled by controlled Poisson processes. market
Source sentence
The mid-price Xt is a continuous-time Markov chain on state space SX with the transition 6 rate matrix (a.k.a. Q-matrix) QX(a) defined as QX(a) := −λ1,2(a) λ1,2(a) 0 0 0 λ2,1(a) −(λ2,1(a) + λ2,3(a)) λ2,3(a) 0 0 0 ... ... ... 0 0 0 ... ... ...
- The optimal policies in the continuous‑time and discretized MDPs are stationary Markov policies. model
Source sentence
Following Kakumanu (1971) and Section 1.7 in Gihman and Skorohod (2012), for the MDP M0 (resp. M∆), if a policy can be expressed as at = π(St) (resp. ai = π(Si)) where π : S →A is a deterministic mapping, then we say this policy is a stationary Markov policy and we denote this policy as π(·). Theorem 1. Under Assumption 1, there exist stationary Markov policies π∗ 0(·) and π∗ ∆(·), such that the optimal value
- The learning rate β(n)(s,a) follows a polynomial decay β = (N(s,a,n))^{-ω} with ω∈(1/2,1). other
Source sentence
Following Even-Dar et al. (2003), we use a polynomial learning rate β(n)(s, a) which is given by β(n)(s, a) = (N(s, a, n))−ω, where ω ∈(1 2, 1) and N(s, a, n) is the one plus the number of times, until the nth iteration, that we visited the state-action pair (s, a).
Show 1 more
- The sample‑complexity analysis uses a definition based on the value‑function error rather than the Q‑function error. other
Source sentence
Our definition of sample complexity is slightly different with that of Even-Dar et al. (2003) in that, we use the value function error while Even-Dar et al. (2003) used the Q function error. Theorem 2.
Data
- Monte Carlo simulation synthetic public
Source sentence
Monte Carlo simulation evidence support all of our theories. To capture the sample complexity using numerical simulations, we record the iteration number N∆, which is the smallest iteration steps n such that ||V (n) ∆(·) −V ∗ ∆(·)|| ≤0.1, when applying the Q-learning algorithm to the discrete-time MDP for each value of ∆.
Costs
Unclear - sources disagree
Source sentence (the authors' words)
The reward functions of our models in Sections 2 and 3.1 do not include the transaction costs due to the convenience of technical analysis.
Finds 4
- A trade‑off exists: decreasing the time increment Δ reduces learning error but increases sample‑complexity (and thus transaction‑cost exposure). theoretical
Source sentence
We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger. Regarding the upper bound εV + CV ∆as a metric of the learning error and the upper bound in (6) as a metric of the sample complexity, we have the tradeoffas
- The sample‑complexity upper bound is polynomial in all model parameters and decreases with Δ, diverging as Δ→0. theoretical
Source sentence
A notable and desirable result in Theorem 2 is that, the sample complexity upper bound is polynomial, rather than exponential, in all model parameters. In (6), the exponents of ∆are both negative, since 6−2 ω −(|SX|+|SY |)(3+ 1 ω) ≤6−2 ω −2(3+ 1 ω) ≤−4 ω < 0 and (1−|SX|−|SY |) 1 1−ω ≤ − 1 1−ω < 0.
- Numerical experiments confirm that the required number of Q‑learning iterations grows as Δ decreases, validating the theoretical sample‑complexity bound. performance
Source sentence
This increasing trend of the iteration numbers demonstrate our theory about the sample complexity, i.e., the sample complexity upper bound will increase when ∆decreases. So, the only difference in the upper bound is the values of ∆and the upward trend in Figure 3 does validate Theorem 2.
- The Nash equilibrium of the discretized two‑player game converges to the continuous‑time Nash equilibrium as Δ→0. theoretical
Source sentence
We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as $Δ\rightarrow0$. A pair of strategies (π1,∗ ∆, π2,∗ ∆) ∈Π1 ∆× Π2 ∆is called a Nash equilibrium if V 1,π1,∗ ∆,π2,∗ ∆ ∆ (s) ≥V 1,π1,π2,∗ ∆ ∆ (s) and V 2,π1,∗ ∆,π2,∗ ∆ ∆ (s) ≥V 2,π1,∗ ∆,π2 ∆ (s), for any (π1, π2) ∈Π1 ∆× Π2 ∆and s ∈S. In Theorem 5 below, we
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 23 related papers). Coverage is partial, so this is neither replication nor refutation.
4 papers in the archive
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (41)
- Robust Market Making: To Quote, or not To Quote — → 2508.16588
- Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2023) — link
- High frequency market making: The role of speed (2023) — link
- Can maker-taker fees prevent algorithmic cooperation in market making? (2022) — link
- AI-driven liquidity provision in OTC financial markets (2022) — link
- Towards multi‐agent reinforcement learning‐driven over‐the‐counter market simulations (2022) — link
- Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality (2022) — link
- Recent advances in reinforcement learning in finance (2021) — link
and 33 more
- Interactions of market making algorithms: a study on perceived collusion (2021) — link
- Reinforcement Learning Approaches to Optimal Market Making (2021) — link
- Towards a fully rl-based market simulator (2021) — link
- Dynamic Equilibrium of Market Making with Price Competition (2020) — link
- Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
- Financial Machine Learning (2019) — link
- Q-learning with Nearest Neighbors (2018) — link
- High-Frequency Financial Econometrics (2016) — link
- Buy Low, Sell High: A High Frequency Trading Perspective (2014) — link
- Dealing with the inventory risk: a solution to the market making problem (2011) — link
- A Stochastic Model for Order Book Dynamics (2010) — link
- High-frequency trading in a limit order book (2008) — link
- Applied stochastic processes and control for jump-diffusions - modeling, analysis, and computation (2007)
- Games with Imperfectly Observable Actions in Continuous Time (2007) — link
- Nonzero-sum games for continuous-time Markov chains with unbounded discounted payoffs (2005) — link
- Learning Rates for Q-learning (2004) — link
- Nash Q-Learning for General-Sum Stochastic Games (2003)
- Competitive Markov Decision Processes (1996) — link
- Controlled markov processes and viscosity solutions (1994) — link
- Towards a Theory of Discounted Repeated Games with Imperfect Monitoring (1990) — link
- Informed Speculation with Imperfect Competition (1989) — link
- Continuous Auctions and Insider Trading (1985) — link
- On the Convergence of the Discrete Time Dynamic Programming Equation for General Semigroups (1982) — link
- Optimal dealer pricing under transactions and return uncertainty (1981) — link
- Dealership market: Market-making with inventory (1980) — link
- Continuously Discounted Markov Decision Model with Countable State and Action Space (1971) — link
- Algorithmic Collusion in Electronic Markets: The Impact of Tick Size (2022) — link
- Sample Complexity Bounds of Exploration (2012) — link
- Reinforcement Learning: An Introduction (2005) — link
- Non-Cooperative Games (2003)
- Market structure, information, futures markets, and price formation. (1984)
- Controlled Stochastic Processes (1979) — link
- Assumption 2
1 of 41 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.