{"paper_id":"2512.04697","title":"Continuous-time reinforcement learning for optimal switching over multiple regimes","authors":["Yijie Huang","Mengge Li","Xiang Yu","Zhou Zhou"],"paper_date":"2025-12-04","paper_type":"theoretical","paper_url":"https://arxiv.org/pdf/2512.04697v2","page_url":"https://thequant.space/flowcharts/continuous-time-reinforcement-learning-for-optimal-switching/","scores":{"empirical_rigor":4,"math_complexity":9.5,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes an exploratory continuous-time reinforcement learning formulation for optimal switching with entropy regularization that randomizes both switching times and regime selection via the generator matrix of a finite-state Markov chain, and devises a reinforcement learning algorithm bas","quote":"Finally, a reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. To 2 this end, we propose a type of exploratory formulation where the decision-maker randomizes both switching time and the selection of the targeted regime state by invoking a generator matrix of an associated continuous-time Markov chain (CTMC) defined on finite state","summary":"This paper studies the continuous-time reinforcement learning (RL) for optimal switching problems across multiple regimes. We consider a type of exploratory formulation under entropy regularization where the agent randomizes both the timing of switches and the selection of regimes through the…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The running profit function f(·,·,i) and terminal reward h(·) are continuous for each regime i.","type":"model","stated":"explicit","quote":"(i) For i ∈Im, the running profit f(·, ·, i) and terminal reward h(·) are assumed to be continuous.","support":"full"},{"assumption":"Both f and h are uniformly bounded by a constant K_{f,h}.","type":"model","stated":"explicit","quote":"Furthermore, there exists a constant Kf,h > 0 such that |f(t, x, i)| + |h(x)| ≤Kf,h, ∀(t, x, i) ∈[0, T] × Rn × Im.","support":"full"},{"assumption":"Switching costs g_{ij} are positive for i≠j (with g_{ii}=0).","type":"model","stated":"explicit","quote":"(2.5) (ii) For i, j ∈Im with j̸ = i, the cost for switching from regime i to j is positive, that is, gij > 0, with the convention gii = 0.","support":"full"},{"assumption":"Direct switching from i to k is cheaper than switching via an intermediate j: g_{ik} < g_{ij}+g_{jk}.","type":"model","stated":"explicit","quote":"For i, j, k ∈Im with j̸ = i, k, it is less expensive to switch directly in one step from regime i to k than in two steps via an intermediate regime j, that is, gik < gij + gjk.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Synthetic switching example","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"The parameters are chosen with symmetry: the drift coefficients are µ0 = −2 and µ1 = 2, and the volatility is σ = 0.5. This symmetric setup induces a natural switching logic to correct the state’s deviation.","support":"full"},{"name":"Investor regime allocation example","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"Here the parameters are set by (µA, σA) = (0.1, 0.2) and (µB, σB) = (0.05, 0.1), and W = (Wt)t∈[0,T] is a standard Brownian motion. The risk free rate is r = 0.05.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":"Switching cost g_{01}=g_{10}=0.5 in the synthetic example; transaction cost matrix G with entries up to 0.02 in the investor example.","quote":"Each switch between regimes incurs a cost, specified as g01 = g10 = 0.5. Switching between regimes incurs transaction costs given by the matrix: G = (gij)0≤i,j≤2 =   0 0.02 0.01 0.02 0 0.01 0.02 0.02 0  .","source":"evidence_card","rule":"R2","candidates":{"front_matter":"unclear","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Switching cost g_{01}=g_{10}=0.5 in the synthetic example; transaction cost matrix G with entries up to 0.02 in the investor example.","card_quote":"Each switch between regimes incurs a cost, specified as g01 = g10 = 0.5. Switching between regimes incurs transaction costs given by the matrix: G = (gij)0≤i,j≤2 =   0 0.02 0.01 0.02 0 0.01 0.02 0.02 0  .","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"The policy iteration method converges with an explicit rate, establishing a new convergence result for the PDE system.","kind":"theoretical","quote":"As the main result of this paper, in the context of PDE system, we establish the convergence result of the policy iteration in Theorem 4.2 with an explicit convergence rate, which is new to the literature.","support":"full"},{"finding":"The exploratory value function converges to the classical optimal switching value function as the temperature parameter λ → 0.","kind":"theoretical","quote":"We also show the convergence of the value function in the exploratory formulation towards the value function in the classical formulation as the temperature parameter vanishes.","support":"full"},{"finding":"Numerical examples using neural networks demonstrate the effectiveness of the proposed RL algorithm in solving the exploratory optimal switching problem.","kind":"performance","quote":"Our numerical examples with the aid of neural networks illustrate the effectiveness of the proposed RL algorithm. We use a discrete version of (6.1) for t = 0, ∆t, ..., K∆t with T = 1, K = 100 and ∆t = T/K. The value function and policy are approximated by a neural network in the PyTorch framework with the architecture and parameters summarized in Table 1.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":12,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:45:18Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}