Paper: arXiv 2404.12598
Authors: Yanwei Jia
Abstract
This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent’s risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (2023) the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton’s investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 3.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper is highly mathematical, featuring advanced stochastic calculus, martingale theory, and convergence proofs for complex continuous-time problems, while the empirical backing is limited to theoretical proofs and simulation experiments on synthetic data without backtesting or real-world datasets.
Research Flowchart
flowchart TD R["Research Goal: Continuous-time risk-sensitive RL with entropy regularization"] M["Key Methodology: Martingale formulation with quadratic variation penalty"] D["Data/Inputs: Merton investment problem & linear-quadratic control simulations"] C["Computational Process: Adapted Q-learning algorithm for risk-sensitive objectives"] O["Outcomes: Proof of convergence, quantitative impact of temperature parameter, improved finite-sample performance"] R --> M M --> D D --> C C --> O
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a continuous-time risk-sensitive reinforcement learning framework that adds a quadratic variation penalty to the value process and introduces a risk-sensitive q‑learning algorithm to incorporate risk sensitivity into existing RL methods.
Source sentence
This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process.
Assumes 1
- The standard assumptions used in ordinary, non‑risk‑sensitive stochastic control problems are adopted. model
Source sentence
I make the standard assumptions in the ordinary, non-risk-sensitive stochastic control problems (Yong and Zhou, 1999).
Data
- Synthetic data for Merton’s investment problem and linear‑quadratic control experiments synthetic unknown
Source sentence
5 Applications I illustrate our methods on two applications with synthetic data sets. The number of episodes within each simulation run is 105.
Costs
Not applicable
Finds 5
- Risk‑sensitive RL improves finite‑sample performance in the linear‑quadratic control problem compared to non‑risk‑sensitive approaches. performance
Source sentence
I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem. With suitable values of ϵ, solving the risk-sensitive objective reduces MSE.
- With suitable values of the risk‑sensitivity coefficient ϵ, the mean‑squared error of the estimated policy is reduced; overly large |ϵ| harms accuracy. statistical
Source sentence
With suitable values of ϵ, solving the risk-sensitive objective reduces MSE. As ϵ approaches 0, the results get closer to the non-risk-sensitive objective, whereas overly large ϵ (in the absolute value) harms the accuracy.
- The optimal choice of ϵ depends on sample size: larger samples favor smaller risk sensitivity. theoretical
Source sentence
Overall, as the sample size increases, the performance get significantly better. The optimal ϵ is between ´5 29 and ´2 when the sample size is small, and is between ´2 and ´1 when the sample size is large.
- The proposed algorithm converges for Merton’s investment problem and the impact of the temperature parameter λ on learning behavior is quantified. theoretical
Source sentence
Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure.
- A smaller temperature parameter λ tends to yield lower estimation error in the off‑policy, offline setting. performance
Source sentence
In addition, a smaller temperature parameter λ seems to induce an overall smaller estimation error in the off-policy, offline setting.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.
2 papers in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (64)
- Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts (2024) — link
- Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning (2024) — link
- Risk-sensitive Markov Decision Process and Learning under General Utility Functions (2023) — link
- Continuous Time q-Learning for Mean-Field Control Problems — → 2306.16208
- Learning equilibrium mean‐variance strategy (2023) — link
- Actor-Critic learning for mean-field control in continuous time (2023) — link
- A Finite Sample Complexity Bound for Distributionally Robust Q-learning (2023) — link
- Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents (2023) — link
and 56 more
- q-Learning in Continuous Time (2022) — link
- The reinforcement learning Kelly strategy (2022) — link
- Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms (2021) — link
- Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning (2021) — link
- Exploratory HJB Equations and Their Convergence (2021) — link
- Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach (2021) — link
- Policy iterations for reinforcement learning problems in continuous time and space - Fundamental theory and methods (2020) — link
- Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls (2020) — link
- Entropy Regularization for Mean Field Games with Learning (2020) — link
- Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret (2020) — link
- A Dynamic Mean-Variance Analysis for Log Returns (2020) — link
- Continuous‐time mean–variance portfolio selection: A reinforcement learning framework (2019) — link
- Making Deep Q-learning methods robust to time discretization (2019) — link
- Is Q-learning Provably Efficient? (2018) — link
- Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances (2018) — link
- Equivalence Between Policy Gradients and Soft Q-Learning (2017) — link
- Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach (2015) — link
- Model-Based Reinforcement Learning in Continuous Environments Using Real-Time Constrained Optimization (2015) — link
- Risk-Sensitive Investment Management (2014) — link
- Reinforcement learning in robotics: A survey (2013) — link
- Robust Portfolio Control with Stochastic Factor Dynamics (2013) — link
- General Bounds and Finite-Time Improvement for the Kiefer-Wolfowitz Stochastic Approximation Algorithm (2011) — link
- Robustness and ambiguity in continuous time (2011) — link
- Remarks on Risk-Sensitive Control Problems (2005) — link
- Robust Portfolio Rules and Asset Pricing (2004) — link
- Robust control and recursive utility (2003) — link
- Q-Learning for Risk-Sensitive Control (2002) — link
- Robust Control and Model Uncertainty (2001) — link
- Risk‐Sensitive Control and an Optimal Investment Model (2000) — link
- Stochastic Controls: Hamiltonian Systems and HJB Equations (1999)
- Risk-Sensitive Dynamic Asset Management (1999) — link
- Robust Properties of Risk-Sensitive Control (1998) — link
- A Stochastic Approximation Algorithm with Varying Bounds (1995) — link
- Risk-Sensitive Control on an Infinite Time Horizon (1995) — link
- Reinforcement learning in continuous time: advantage updating (1994) — link
- On the existence of optimal relaxed controls of stochastic partial differential equations (1992) — link
- Stochastic differential utility (1992) — link
- Substitution, Risk Aversion, and the Temporal Behavior of Consumption and Asset Returns: A Theoretical Framework (1989) — link
- Knight on Risk and Uncertainty (1987) — link
- On stochastic relaxed control for partially observed diffusions (1984) — link
- Asymptotic evaluation of certain Markov process expectations for large time (1975) — link
- Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games (1973) — link
- Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case (1969) — link
- A Stochastic Approximation Method (1951) — link
- Reinforcement Learning for Continuous-Time Optimal Execution: Actor-Critic Algorithm and Error Analysis (2023) — link
- Reinforcement Learning in Continuous Time and Space: A Stochastic Control Approach (2020)
- Continuous Martingales and Brownian Motion , volume 293 (2013)
- The exact law of large numbers via Fubini extension and characterization of insurable risks (2006) — link
- Reinforcement Learning in Continuous Time and Space (2000) — link
- Bellman Equations of Risk-Sensitive Control (1996) — link
- Maxmin Expected Utility with Non-Unique Prior (1989) — link
- A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications (1985) — link
- Stochastic Approximation (1969) — link
- If ǫ ă 0 , t Z s ` ǫ 2 x Z yp s q , s ě 0 u is an pt F s u s ě 0 , P q -local submartingale (supermartingale), then t e ǫZ s , s ě 0 is also a F s s 0 , P supermartingale (submartingale)
- ii) the SDE (6) admits a unique weak solution (in the sense of distribution) for any initial p t, x q P 0
- Learning Merton’s strategies in an incomplete market: Recursive entropy regularization and biased Gaussian exploration
1 of 64 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.