Paper: arXiv 2512.04697
Authors: Yijie Huang, Mengge Li, Xiang Yu, Zhou Zhou
Abstract
This paper studies the continuous-time reinforcement learning (RL) for optimal switching problems across multiple regimes. We consider a type of exploratory formulation under entropy regularization where the agent randomizes both the timing of switches and the selection of regimes through the generator matrix of an associated continuous-time finite-state Markov chain. We establish the well-posedness of the associated system of Hamilton-Jacobi-Bellman (HJB) equations and provide a characterization of the optimal policy. The policy improvement and the convergence of the policy iterations are rigorously established by analyzing the system of equations. We also show the convergence of the value function in the exploratory formulation towards the value function in the classical formulation as the temperature parameter vanishes. Finally, a reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. Our numerical examples with the aid of neural networks illustrate the effectiveness of the proposed RL algorithm.
Complexity vs Empirical Score
- Math Complexity: 9.5/10
- Empirical Rigor: 4.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper is mathematically dense, featuring advanced stochastic analysis, systems of Hamilton-Jacobi-Bellman (HJB) equations, and rigorous convergence proofs. While it includes a numerical implementation with neural networks, the core contribution is theoretical with analytical proofs dominating the rigor, focusing on PDE existence and convergence rather than extensive backtesting on real financial datasets.
Research Flowchart
flowchart TD
A["Research Goal<br>Continuous-time RL for Optimal Switching"] --> B["Methodology: Entropy-Regularized Formulation"]
B --> C{"Analytical Derivation"}
C --> D["HJB Equations &<br>Optimal Policy Characterization"]
C --> E["Convergence Analysis<br>Exploratory → Classical Value"]
B --> F["Algorithm Design<br>RL with Policy Evaluation"]
F --> G["Computational Process<br>Neural Network Implementation"]
D & E & G --> H["Outcomes<br>Validated RL Algorithm &<br>Theoretical Guarantees"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes an exploratory continuous-time reinforcement learning formulation for optimal switching with entropy regularization that randomizes both switching times and regime selection via the generator matrix of a finite-state Markov chain, and devises a reinforcement learning algorithm bas
Source sentence
Finally, a reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. To 2 this end, we propose a type of exploratory formulation where the decision-maker randomizes both switching time and the selection of the targeted regime state by invoking a generator matrix of an associated continuous-time Markov chain (CTMC) defined on finite state
Assumes 4
- The running profit function f(·,·,i) and terminal reward h(·) are continuous for each regime i. model
Source sentence
(i) For i ∈Im, the running profit f(·, ·, i) and terminal reward h(·) are assumed to be continuous.
- Both f and h are uniformly bounded by a constant K_{f,h}. model
Source sentence
Furthermore, there exists a constant Kf,h > 0 such that |f(t, x, i)| + |h(x)| ≤Kf,h, ∀(t, x, i) ∈[0, T] × Rn × Im.
- Switching costs g_{ij} are positive for i≠j (with g_{ii}=0). model
Source sentence
(2.5) (ii) For i, j ∈Im with j̸ = i, the cost for switching from regime i to j is positive, that is, gij > 0, with the convention gii = 0.
- Direct switching from i to k is cheaper than switching via an intermediate j: g_{ik} < g_{ij}+g_{jk}. model
Source sentence
For i, j, k ∈Im with j̸ = i, k, it is less expensive to switch directly in one step from regime i to k than in two steps via an intermediate regime j, that is, gik < gij + gjk.
Data
- Synthetic switching example synthetic unknown
Source sentence
The parameters are chosen with symmetry: the drift coefficients are µ0 = −2 and µ1 = 2, and the volatility is σ = 0.5. This symmetric setup induces a natural switching logic to correct the state’s deviation.
- Investor regime allocation example synthetic unknown
Source sentence
Here the parameters are set by (µA, σA) = (0.1, 0.2) and (µB, σB) = (0.05, 0.1), and W = (Wt)t∈[0,T] is a standard Brownian motion. The risk free rate is r = 0.05.
Costs
Modeled
Switching cost g_{01}=g_{10}=0.5 in the synthetic example; transaction cost matrix G with entries up to 0.02 in the investor example.
Source sentence
Each switch between regimes incurs a cost, specified as g01 = g10 = 0.5. Switching between regimes incurs transaction costs given by the matrix: G = (gij)0≤i,j≤2 = 0 0.02 0.01 0.02 0 0.01 0.02 0.02 0 .
Finds 3
- The policy iteration method converges with an explicit rate, establishing a new convergence result for the PDE system. theoretical
Source sentence
As the main result of this paper, in the context of PDE system, we establish the convergence result of the policy iteration in Theorem 4.2 with an explicit convergence rate, which is new to the literature.
- The exploratory value function converges to the classical optimal switching value function as the temperature parameter λ → 0. theoretical
Source sentence
We also show the convergence of the value function in the exploratory formulation towards the value function in the classical formulation as the temperature parameter vanishes.
- Numerical examples using neural networks demonstrate the effectiveness of the proposed RL algorithm in solving the exploratory optimal switching problem. performance
Source sentence
Our numerical examples with the aid of neural networks illustrate the effectiveness of the proposed RL algorithm. We use a discrete version of (6.1) for t = 0, ∆t, ..., K∆t with T = 1, K = 100 and ∆t = T/K. The value function and policy are approximated by a neural network in the PyTorch framework with the architecture and parameters summarized in Table 1.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 12 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (39)
- A stochastic approximation method (1951)
- Hybrid Switching Diffusions: Properties and Applications (2025)
- Continuous time q-learning for mean-field control problems — → 2306.16208
- On optimal tracking portfolio in incomplete markets: T he reinforcement learning approach (2025)
- Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market (2024)
- A reinforcement learning framework for some singular stochastic control problems (2025) — link
- Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach (2025) — link
- Continuous-time q-learning for jump-diffusion models under T sallis entropy (2026)
and 31 more
- Reinforcement learning for jump-diffusions, with financial applications — → 2405.16449
- Exploratory optimal stopping: A singular control formulation — → 2408.09335
- Learning to optimally stop diffusion processes, with financial applications — → 2408.09242
- Exploratory HJB equations and their convergence (2022)
- Policy iteration for exploratory H amilton– J acobi– B ellman equations (2025)
- Convergence of policy iteration for entropy-regularized stochastic control problems (2025)
- q- L earning in continuous time (2023)
- Policy gradient and actor-critic learning in continuous time and space: T heory and algorithms (2022)
- Reinforcement learning for arbitrage strategies in stock index futures (2025)
- A two-fold randomization framework for impulse control problems (2025) — link
- Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching (2025)
- Reinforcement learning in continuous time and space: A stochastic control approach (2020)
- Valuation of energy storage: A n optimal switching approach (2010)
- Valuation of a power plant under production constraints and market incompleteness (2009)
- Management strategies for run-of-river hydropower plants: an optimal switching approach (2022)
- Pricing asset scheduling flexibility using optimal switching (2008)
- Adaptive algorithms and stochastic approximations (2012)
- A stochastic target formulation for optimal switching problems in finite horizon (2009)
- Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
- Optimal control and viscosity solutions of H amilton- J acobi- B ellman equations (1997)
- Stochastic optimal multi-modes switching with a viscosity solution approach (2013)
- On the first boundary problem for quasilinear systems of parabolic differential equations in non-cylindrical domains (1965)
- Continuous-time stochastic control and optimization with financial applications (2009)
- Explicit solution to an optimal switching problem in the two‐regime case (2007) — link
- Exploratory optimal stopping: A singular control formulation — → 2408.09335
- Randomized optimal stopping problem in continuous time and reinforcement learning algorithm (2024)
- Policy gradient and actor-critic learning in continuous time and space: T heory and algorithms (2022)
- Unified continuous-time q-learning for mean-field game and mean-field control problems — → 2407.04521
- Regret of exploratory policy improvement and q-learning (2024) — link
- Continuous-time q-learning for mean-field control with common noise, part-I: theoretical foundations (2026) — link
- Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms (2026) — link
6 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.