{"paper_id":"2407.21025","title":"Reinforcement Learning in High-frequency Market Making","authors":["Yuheng Zheng","Zihan Ding"],"paper_date":"2024-07-14","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2407.21025v2","page_url":"https://thequant.space/flowcharts/reinforcement-learning-in-high-frequency-market-making/","scores":{"empirical_rigor":6,"math_complexity":9,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper provides a comprehensive theoretical analysis of reinforcement learning for high‑frequency market making, including convergence of discretized models to continuous‑time limits, sample‑complexity upper bounds, and a model‑free Nash Q‑learning algorithm for multi‑agent equilibrium.","quote":"This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger.","summary":"This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We bridge the modern RL theory and the continuous-time statistical models in high-frequency financial economics. Different with most existing…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The transition rate functions λ_{k,l}(a) are bounded away from zero and above by a constant C_λ > 0 for all states and actions.","type":"model","stated":"explicit","quote":"We propose the following assumption on the rate function λk,l. Assumption 1. For the mid-price, the transition rate functions λk,l(a) satisfy that, there exists a constant Cλ > 0 such that 0 < λk,l(a) < Cλ for any k, l and a ∈A. Denote by N a t (resp. N b t ) a controlled Poisson process with intensity λ(|pa t −Xt|) (resp. λ(|pb t − Xt|)), which models the coming ﬂow of the market buy (resp. sell) order at time t.","support":"full"},{"assumption":"The mid‑price dynamics follow a continuous‑time Markov chain with a (possibly action‑dependent) Q‑matrix, and market order flow is modeled by controlled Poisson processes.","type":"market","stated":"explicit","quote":"The mid-price Xt is a continuous-time Markov chain on state space SX with the transition 6 rate matrix (a.k.a. Q-matrix) QX(a) deﬁned as QX(a) :=   −λ1,2(a) λ1,2(a) 0 0 0 λ2,1(a) −(λ2,1(a) + λ2,3(a)) λ2,3(a) 0 0 0 ... ... ... 0 0 0 ... ... ...","support":"full"},{"assumption":"The optimal policies in the continuous‑time and discretized MDPs are stationary Markov policies.","type":"model","stated":"explicit","quote":"Following Kakumanu (1971) and Section 1.7 in Gihman and Skorohod (2012), for the MDP M0 (resp. M∆), if a policy can be expressed as at = π(St) (resp. ai = π(Si)) where π : S →A is a deterministic mapping, then we say this policy is a stationary Markov policy and we denote this policy as π(·). Theorem 1. Under Assumption 1, there exist stationary Markov policies π∗ 0(·) and π∗ ∆(·), such that the optimal value","support":"full"},{"assumption":"The learning rate β(n)(s,a) follows a polynomial decay β = (N(s,a,n))^{-ω} with ω∈(1/2,1).","type":"other","stated":"explicit","quote":"Following Even-Dar et al. (2003), we use a polynomial learning rate β(n)(s, a) which is given by β(n)(s, a) = (N(s, a, n))−ω, where ω ∈(1 2, 1) and N(s, a, n) is the one plus the number of times, until the nth iteration, that we visited the state-action pair (s, a).","support":"full"},{"assumption":"The sample‑complexity analysis uses a definition based on the value‑function error rather than the Q‑function error.","type":"other","stated":"explicit","quote":"Our deﬁnition of sample complexity is slightly diﬀerent with that of Even-Dar et al. (2003) in that, we use the value function error while Even-Dar et al. (2003) used the Q function error. Theorem 2.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Monte Carlo simulation","provider":null,"kind":"synthetic","access":"public","period":null,"universe":null,"quote":"Monte Carlo simulation evidence support all of our theories. To capture the sample complexity using numerical simulations, we record the iteration number N∆, which is the smallest iteration steps n such that ||V (n) ∆(·) −V ∗ ∆(·)|| ≤0.1, when applying the Q-learning algorithm to the discrete-time MDP for each value of ∆.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"available","modeled":"unclear","headline_net_of_costs":null,"detail":null,"quote":null,"source":"conflict","rule":"R4b","candidates":{"front_matter":"n/a","evidence_card":"no","evidence_card_quoted":true},"conflict_kind":"applicability","card_detail":null,"card_quote":"The reward functions of our models in Sections 2 and 3.1 do not include the transaction costs due to the convenience of technical analysis.","meaning":"the platform classifier and the paper-text extraction disagree, so no value is asserted"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"A trade‑off exists: decreasing the time increment Δ reduces learning error but increases sample‑complexity (and thus transaction‑cost exposure).","kind":"theoretical","quote":"We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger. Regarding the upper bound εV + CV ∆as a metric of the learning error and the upper bound in (6) as a metric of the sample complexity, we have the tradeoﬀas","support":"full"},{"finding":"The sample‑complexity upper bound is polynomial in all model parameters and decreases with Δ, diverging as Δ→0.","kind":"theoretical","quote":"A notable and desirable result in Theorem 2 is that, the sample complexity upper bound is polynomial, rather than exponential, in all model parameters. In (6), the exponents of ∆are both negative, since 6−2 ω −(|SX|+|SY |)(3+ 1 ω) ≤6−2 ω −2(3+ 1 ω) ≤−4 ω < 0 and (1−|SX|−|SY |) 1 1−ω ≤ − 1 1−ω < 0.","support":"full"},{"finding":"Numerical experiments confirm that the required number of Q‑learning iterations grows as Δ decreases, validating the theoretical sample‑complexity bound.","kind":"performance","quote":"This increasing trend of the iteration numbers demonstrate our theory about the sample complexity, i.e., the sample complexity upper bound will increase when ∆decreases. So, the only diﬀerence in the upper bound is the values of ∆and the upward trend in Figure 3 does validate Theorem 2.","support":"full"},{"finding":"The Nash equilibrium of the discretized two‑player game converges to the continuous‑time Nash equilibrium as Δ→0.","kind":"theoretical","quote":"We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as $Δ\\rightarrow0$. A pair of strategies (π1,∗ ∆, π2,∗ ∆) ∈Π1 ∆× Π2 ∆is called a Nash equilibrium if V 1,π1,∗ ∆,π2,∗ ∆ ∆ (s) ≥V 1,π1,π2,∗ ∆ ∆ (s) and V 2,π1,∗ ∆,π2,∗ ∆ ∆ (s) ≥V 2,π1,∗ ∆,π2 ∆ (s), for any (π1, π2) ∈Π1 ∆× Π2 ∆and s ∈S. In Theorem 5 below, we","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":23,"citations":{"cited_by_in_corpus":4,"by_independent_authors":4,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":1,"unclassified":3},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/zyh-pku/Reinforcement-Learning-and-Market-Making","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:56:28Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}