{"paper_id":"2306.16208","title":"Continuous-time q-learning for mean-field control problems","authors":["Xiaoli Wei","Xiang Yu"],"paper_date":"2023-06-28","paper_type":"methodological","paper_url":"https://arxiv.org/pdf/2306.16208v4","page_url":"https://thequant.space/flowcharts/continuous-time-q-learning-for-mean-field-control-problems/","scores":{"empirical_rigor":6.8,"math_complexity":9.2,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper devises model‑free continuous‑time q‑learning algorithms for mean‑field control problems by introducing two q‑functions (integrated q and essential q) and using a weak martingale condition with test policies.","quote":"Based on the weak martingale condition and our proposed searching method of test policies, some model-free learning algorithms are devised. In two examples, one in LQ control framework and one beyond LQ control framework, we can obtain the exact parameterization of the optimal value function and q-functions and illustrate our algorithms with simulation experiments.","summary":"This paper studies the q-learning, recently coined as the continuous time counterpart of Q-learning by Jia and Zhou (2023), for continuous time Mckean-Vlasov control problems in the setting of entropy-regularized reinforcement learning. In contrast to the single agent’s control problem in Jia and…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The coefficients bπ and σπ are differentiable in x and µ with bounded, locally Lipschitz derivatives up to second order.","type":"model","stated":"explicit","quote":"6 Assumption 2.1 (i) For f ∈{bπ, σπ}, the derivatives ∂xf(t, x, µ), ∂2 xf(t, x, µ), ∂µf(t, x, µ)(v) and ∂v∂µf(t, x, µ)(v) exist for any (t, x, v, µ) ∈[0, T] × Rd × Rd × P2(Rd), are bounded and locally Lipschitz continuous with respect to x, µ, v uniformly in t ∈[0, T].","support":"full"},{"assumption":"The functions bπ and σπ satisfy linear growth: |f(t,x,µ)| ≤ C(1+|x|+M₂(µ)).","type":"model","stated":"explicit","quote":"For all (t, x, µ) ∈[0, T] × Rd × P2(Rd), we have |f(t, x, µ)| ≤C(1 + |x| + M2(µ)).","support":"full"},{"assumption":"The reward functions rπ(t,·), entropy term Eπ(t,·) and terminal reward g are twice continuously differentiable in (x,µ).","type":"model","stated":"explicit","quote":"(ii) For any t ∈[0, T], rπ(t, ·), Eπ(t, ·) and g(·) ∈C2,2(Rd × P2(Rd)).","support":"full"},{"assumption":"The reward, entropy and terminal functions have at most quadratic growth in x and µ and their first‑order derivatives are bounded by C(1+|x|+M₂(µ)).","type":"model","stated":"explicit","quote":"(iii) There exists some constant C < ∞, such that for any (t, x, v, µ) ∈[0, T]×Rd×Rd×P2(Rd), we have |rπ(t, x, µ)| + |Eπ(t, x, µ)| + |g(x, µ)| ≤C(1 + |x|2 + M2(µ)), |∂xrπ(t, x, µ)| + |∂xEπ(t, x, µ)| + |∂xg(x, µ)| ≤C(1 + |x| + M2(µ)), |∂µrπ(t, x, µ)(v)| + |∂µEπ(t, x, µ)(v)| + |∂µg(x, µ)(v)| ≤C(1 + |x| + |v| + M2(µ)), |∂v∂µrπ(t, x, µ)(v)| + |∂2 xrπ(t, x, µ)| + |∂v∂µEπ(t, x, µ)(v)| + |∂2 xEπ(t, x, µ)| + |∂v∂µg(x,","support":"full"},{"assumption":"Higher‑order derivatives of rπ, Eπ and g with respect to x, µ and v are bounded by C(1+M₂(µ)).","type":"model","stated":"explicit","quote":"µ)| ≤C(1 + M2(µ)).","support":"full"},{"assumption":"Under the above regularity, the value function J(t,µ;π) is C¹,² on [0,T]×P₂(ℝᵈ).","type":"model","stated":"explicit","quote":"Under Assumption 2.1, Proposition 2.1 of Frikha et al. (2023) guarantees that the function J(t, µ; π) defined in (2.5) is of C1,2([0, T] × P2(Rd)).","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Synthetic LQ control example","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"In two examples, one in LQ control framework and one beyond LQ control framework, we can obtain the exact parameterization of the optimal value function and q-functions and illustrate our algorithms with simulation experiments.","support":"full"},{"name":"Synthetic non‑LQ control example","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"In two examples, one in LQ control framework and one beyond LQ control framework, we can obtain the exact parameterization of the optimal value function and q-functions and illustrate our algorithms with simulation experiments.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"Exact parameterizations of the optimal value function and both q‑functions are obtained for the LQ and a non‑LQ example.","kind":"theoretical","quote":"In two examples, one in LQ control framework and one beyond LQ control framework, we can obtain the exact parameterization of the optimal value function and q-functions and illustrate our algorithms with simulation experiments.","support":"full"},{"finding":"Simulation experiments illustrate that the proposed model‑free algorithms successfully learn the integrated and essential q‑functions.","kind":"performance","quote":"In two examples, one in LQ control framework and one beyond LQ control framework, we can obtain the exact parameterization of the optimal value function and q-functions and illustrate our algorithms with simulation experiments. In two examples, one in the LQ control framework and one beyond the LQ control framework, we can illustrate the effectiveness of our q-learning algorithm.","support":"full"},{"finding":"A weak martingale characterization of the value function and integrated q‑function is established using test policies in the neighbourhood of the target policy.","kind":"theoretical","quote":"Comparing with two counterparts in Gu et al. (2023) and Jia and Zhou (2023), we establish the weak martingale characterization of the value function and the integrated q-function through test polices in the neighbourhood of the target policy in a similar flavor of 30 off-policy learning.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":11,"citations":{"cited_by_in_corpus":4,"by_independent_authors":3,"how_cited":{"used_as_baseline":0,"uses_method_or_data":1,"extends":0,"discusses_findings":0,"mentions":1,"unclassified":2},"independent_papers":[{"paper":{"paper_id":"2404.12598","title":"Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty","paper_date":"2024-04-19","authors":["Yanwei Jia"],"empirical_rigor":3,"page_url":"https://thequant.space/flowcharts/continuous-time-risk-sensitive-reinforcement-learning-via-qu/"},"how":"uses_method_or_data","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T20:02:22Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}