Paper: arXiv 2404.12598

Authors: Yanwei Jia

Abstract

This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent’s risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (2023) the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton’s investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper is highly mathematical, featuring advanced stochastic calculus, martingale theory, and convergence proofs for complex continuous-time problems, while the empirical backing is limited to theoretical proofs and simulation experiments on synthetic data without backtesting or real-world datasets.

Research Flowchart

  flowchart TD
  R["Research Goal: Continuous-time risk-sensitive RL with entropy regularization"]
  M["Key Methodology: Martingale formulation with quadratic variation penalty"]
  D["Data/Inputs: Merton investment problem & linear-quadratic control simulations"]
  C["Computational Process: Adapted Q-learning algorithm for risk-sensitive objectives"]
  O["Outcomes: Proof of convergence, quantitative impact of temperature parameter, improved finite-sample performance"]
  R --> M
  M --> D
  D --> C
  C --> O

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a continuous-time risk-sensitive reinforcement learning framework that adds a quadratic variation penalty to the value process and introduces a risk-sensitive q‑learning algorithm to incorporate risk sensitivity into existing RL methods.

Source sentence
This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process.

Assumes 1

  • The standard assumptions used in ordinary, non‑risk‑sensitive stochastic control problems are adopted. model
    Source sentence
    I make the standard assumptions in the ordinary, non-risk-sensitive stochastic control problems (Yong and Zhou, 1999).

Data

  • Synthetic data for Merton’s investment problem and linear‑quadratic control experiments synthetic unknown
    Source sentence
    5 Applications I illustrate our methods on two applications with synthetic data sets. The number of episodes within each simulation run is 105.

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • Risk‑sensitive RL improves finite‑sample performance in the linear‑quadratic control problem compared to non‑risk‑sensitive approaches. performance
    Source sentence
    I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem. With suitable values of ϵ, solving the risk-sensitive objective reduces MSE.
  • With suitable values of the risk‑sensitivity coefficient ϵ, the mean‑squared error of the estimated policy is reduced; overly large |ϵ| harms accuracy. statistical
    Source sentence
    With suitable values of ϵ, solving the risk-sensitive objective reduces MSE. As ϵ approaches 0, the results get closer to the non-risk-sensitive objective, whereas overly large ϵ (in the absolute value) harms the accuracy.
  • The optimal choice of ϵ depends on sample size: larger samples favor smaller risk sensitivity. theoretical
    Source sentence
    Overall, as the sample size increases, the performance get significantly better. The optimal ϵ is between ´5 29 and ´2 when the sample size is small, and is between ´2 and ´1 when the sample size is large.
  • The proposed algorithm converges for Merton’s investment problem and the impact of the temperature parameter λ on learning behavior is quantified. theoretical
    Source sentence
    Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure.
  • A smaller temperature parameter λ tends to yield lower estimation error in the off‑policy, offline setting. performance
    Source sentence
    In addition, a smaller temperature parameter λ seems to induce an overall smaller estimation error in the off-policy, offline setting.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 2 papers in the archive (2 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (64)

  1. Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts (2024) — link
  2. Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning (2024) — link
  3. Risk-sensitive Markov Decision Process and Learning under General Utility Functions (2023) — link
  4. Continuous Time q-Learning for Mean-Field Control Problems — → 2306.16208
  5. Learning equilibrium mean‐variance strategy (2023) — link
  6. Actor-Critic learning for mean-field control in continuous time (2023) — link
  7. A Finite Sample Complexity Bound for Distributionally Robust Q-learning (2023) — link
  8. Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents (2023) — link
and 56 more
  1. q-Learning in Continuous Time (2022) — link
  2. The reinforcement learning Kelly strategy (2022) — link
  3. Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms (2021) — link
  4. Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning (2021) — link
  5. Exploratory HJB Equations and Their Convergence (2021) — link
  6. Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach (2021) — link
  7. Policy iterations for reinforcement learning problems in continuous time and space - Fundamental theory and methods (2020) — link
  8. Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls (2020) — link
  9. Entropy Regularization for Mean Field Games with Learning (2020) — link
  10. Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret (2020) — link
  11. A Dynamic Mean-Variance Analysis for Log Returns (2020) — link
  12. Continuous‐time mean–variance portfolio selection: A reinforcement learning framework (2019) — link
  13. Making Deep Q-learning methods robust to time discretization (2019) — link
  14. Is Q-learning Provably Efficient? (2018) — link
  15. Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances (2018) — link
  16. Equivalence Between Policy Gradients and Soft Q-Learning (2017) — link
  17. Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach (2015) — link
  18. Model-Based Reinforcement Learning in Continuous Environments Using Real-Time Constrained Optimization (2015) — link
  19. Risk-Sensitive Investment Management (2014) — link
  20. Reinforcement learning in robotics: A survey (2013) — link
  21. Robust Portfolio Control with Stochastic Factor Dynamics (2013) — link
  22. General Bounds and Finite-Time Improvement for the Kiefer-Wolfowitz Stochastic Approximation Algorithm (2011) — link
  23. Robustness and ambiguity in continuous time (2011) — link
  24. Remarks on Risk-Sensitive Control Problems (2005) — link
  25. Robust Portfolio Rules and Asset Pricing (2004) — link
  26. Robust control and recursive utility (2003) — link
  27. Q-Learning for Risk-Sensitive Control (2002) — link
  28. Robust Control and Model Uncertainty (2001) — link
  29. Risk‐Sensitive Control and an Optimal Investment Model (2000) — link
  30. Stochastic Controls: Hamiltonian Systems and HJB Equations (1999)
  31. Risk-Sensitive Dynamic Asset Management (1999) — link
  32. Robust Properties of Risk-Sensitive Control (1998) — link
  33. A Stochastic Approximation Algorithm with Varying Bounds (1995) — link
  34. Risk-Sensitive Control on an Infinite Time Horizon (1995) — link
  35. Reinforcement learning in continuous time: advantage updating (1994) — link
  36. On the existence of optimal relaxed controls of stochastic partial differential equations (1992) — link
  37. Stochastic differential utility (1992) — link
  38. Substitution, Risk Aversion, and the Temporal Behavior of Consumption and Asset Returns: A Theoretical Framework (1989) — link
  39. Knight on Risk and Uncertainty (1987) — link
  40. On stochastic relaxed control for partially observed diffusions (1984) — link
  41. Asymptotic evaluation of certain Markov process expectations for large time (1975) — link
  42. Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games (1973) — link
  43. Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case (1969) — link
  44. A Stochastic Approximation Method (1951) — link
  45. Reinforcement Learning for Continuous-Time Optimal Execution: Actor-Critic Algorithm and Error Analysis (2023) — link
  46. Reinforcement Learning in Continuous Time and Space: A Stochastic Control Approach (2020)
  47. Continuous Martingales and Brownian Motion , volume 293 (2013)
  48. The exact law of large numbers via Fubini extension and characterization of insurable risks (2006) — link
  49. Reinforcement Learning in Continuous Time and Space (2000) — link
  50. Bellman Equations of Risk-Sensitive Control (1996) — link
  51. Maxmin Expected Utility with Non-Unique Prior (1989) — link
  52. A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications (1985) — link
  53. Stochastic Approximation (1969) — link
  54. If ǫ ă 0 , t Z s ` ǫ 2 x Z yp s q , s ě 0 u is an pt F s u s ě 0 , P q -local submartingale (supermartingale), then t e ǫZ s , s ě 0 is also a F s s 0 , P supermartingale (submartingale)
  55. ii) the SDE (6) admits a unique weak solution (in the sense of distribution) for any initial p t, x q P 0
  56. Learning Merton’s strategies in an incomplete market: Recursive entropy regularization and biased Gaussian exploration

1 of 64 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.