Paper: arXiv 2512.24580

Authors: Shanyu Han, Yangbo He, Yang Liu

Abstract

We propose a novel framework for risk-sensitive reinforcement learning (RSRL) that incorporates robustness against transition uncertainty. We define two distinct yet coupled risk measures: an inner risk measure addressing state and cost randomness and an outer risk measure capturing transition dynamics uncertainty. Our framework unifies and generalizes most existing RL frameworks by permitting general coherent risk measures for both inner and outer risk measures. Within this framework, we construct a risk-sensitive robust Markov decision process (RSRMDP), derive its Bellman equation, and provide error analysis under a given posterior distribution. We further develop a Bayesian Dynamic Programming (Bayesian DP) algorithm that alternates between posterior updates and value iteration. The approach employs an estimator for the risk-based Bellman operator that combines Monte Carlo sampling with convex optimization, for which we prove strong consistency guarantees. Furthermore, we demonstrate that the algorithm converges to a near-optimal policy in the training environment and analyze both the sample complexity and the computational complexity under the Dirichlet posterior and CVaR. Finally, we validate our approach through two numerical experiments. The results exhibit excellent convergence properties while providing intuitive demonstrations of its advantages in both risk-sensitivity and robustness. Empirically, we further demonstrate the advantages of the proposed algorithm through an application on option hedging.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a mathematically rigorous framework unifying risk-sensitivity and model robustness via double-layered coherent risk measures and Bayesian Dynamic Programming. While theoretical foundations, convergence proofs, and sample complexity analyses are strong, the empirical evaluation focuses primarily on stylized numerical examples and a specific option hedging application.

Research Flowchart

  flowchart TD
    A[<b>Research Goal:</b><br>Develop Robust Bayesian DP for<br>Risk-Sensitive Reinforcement Learning] --> B[<b>Framework Formulation:</b><br>Construct RSRMDP with coupled Inner &<br>Outer Coherent Risk Measures]
    
    C[<b>Inputs & Data:</b><br>Environment Transitions,<br>Dirichlet Posterior & CVaR] --> D
    
    B --> D[<b>Core Algorithm:</b><br>Bayesian Dynamic Programming<br><i>(Posterior Updates + Value Iteration)</i>]
    
    D --> E[<b>Computation:</b><br>Monte Carlo Sampling +<br>Convex Optimization Operator]
    
    E --> F[<b>Theoretical Guarantees:</b><br>Strong Consistency, Near-Optimal<br>Policy & Complexity Bounds]
    
    F --> G[<b>Key Outcomes:</b><br>Robustness to Uncertainty &<br>Superior Option Hedging Performance]

Cited References (84)

  1. Robust solutions of optimization problems affected by uncertain probabilities (2013)
  2. Distributionally Robust $ Q $-Learning (2022)
  3. Kullback-Leibler divergence constrained distributionally robust optimization (2013)
  4. Large-scale methods for distributionally robust optimization (2020)
  5. Probability theory: a comprehensive course (2013)
  6. Risk-averse dynamic programming for Markov decision processes (2010)
  7. Bayesian risk-averse q-learning with streaming observations (2023)
  8. Markov decision processes: discrete stochastic dynamic programming (2014)
and 76 more
  1. A distributional perspective on reinforcement learning (2017)
  2. Risk-sensitive and robust decision-making: a cvar optimization approach (2015)
  3. Reinforcement learning: An introduction (1998)
  4. Bayesian dynamic programming (1975)
  5. Learning from delayed rewards (1989)
  6. Bayesian reinforcement learning: A survey (2015)
  7. A Bayesian framework for reinforcement learning (2000)
  8. Coherent risk measures on general probability spaces (2002)
  9. Sequential decision making with coherent risk (2016)
  10. Reinforcement learning with dynamic convex risk measures (2024)
  11. A Bayesian sampling approach to exploration in reinforcement learning (2012) — link
  12. An analytic solution to discrete Bayesian reinforcement learning (2006)
  13. (More) efficient reinforcement learning via posterior sampling (2013)
  14. Equitable partitioning policies for robotic networks (2009)
  15. Optimal transport: discretization and algorithms (2021)
  16. Solving semi-discrete optimal transport problems: star shapedeness and Newton’s method (2024)
  17. Optimally solving a transportation problem using Voronoi diagrams (2013)
  18. A framework for measures of risk under uncertainty (2024)
  19. Convolution Bounds on Quantile Aggregation (2025)
  20. Coherent risk measures in inventory problems (2007)
  21. Coherent risk measures and good-deal bounds (2001)
  22. Bayesian risk Markov decision processes (2022)
  23. Markov decision processes with average-value-at-risk criteria (2011)
  24. Policy gradient based entropic-var optimization in risk-sensitive reinforcement learning (2022)
  25. Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions — → 2505.04553
  26. Risk-sensitive rl with optimized certainty equivalents via reduction to standard rl (2024) — link
  27. Robust Reinforcement Learning with Dynamic Distortion Risk Measures — → 2409.10096
  28. Risk-sensitive reinforcement learning (2014)
  29. Risk-sensitive reinforcement learning (2002)
  30. Conditionally elicitable dynamic risk measures for deep reinforcement learning (2023)
  31. Regret bounds for risk-sensitive reinforcement learning with lipschitz dynamic risk measures (2024)
  32. Risk-averse reinforcement learning via dynamic time-consistent risk measures (2022)
  33. Deep reinforcement learning for option pricing and hedging under dynamic expectile risk measures (2023)
  34. Risk-Sensitive Reinforcement Learning with $\varphi$-Divergence-Risk (2024)
  35. Mean-variance policy iteration for risk-averse reinforcement learning (2021)
  36. A Bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics (2018)
  37. Simulation optimization when facing input uncertainty (2015)
  38. Robust Q-learning algorithm for Markov decision processes under Wasserstein uncertainty (2024)
  39. Natural actor-critic for robust reinforcement learning with function approximation (2023)
  40. Model-free robust average-reward reinforcement learning (2023)
  41. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage (2023)
  42. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2017)
  43. Learning policies for adaptive tracking with deep feature cascades (2017)
  44. Deep direct reinforcement learning for financial signal representation and trading (2016)
  45. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play (2018)
  46. Deep reinforcement learning for option replication and hedging (2020)
  47. Safe learning in robotics: From learning-based control to safe reinforcement learning (2022)
  48. Robust risk-aware reinforcement learning (2022)
  49. Risk-averse model uncertainty for distributionally robust safe reinforcement learning (2023)
  50. Risk-averse bayes-adaptive reinforcement learning (2021)
  51. Robust and adaptive planning under model uncertainty (2019)
  52. Risk averse robust adversarial reinforcement learning (2019)
  53. Q-learning (1992)
  54. Sur les op 'e rations dans les ensembles abstraits et leur application aux 'e quations int 'e grales (1922)
  55. Asymptotics and special functions (1997)
  56. Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk (2024)
  57. Soft robust MDPs and risk-sensitive MDPs: Equivalence, policy gradient, and sample complexity (2023) — link
  58. Robustness and risk-sensitivity in Markov decision processes (2012)
  59. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity (2024)
  60. Robust reinforcement learning using least squares policy iteration with provable performance guarantees (2021)
  61. Provable risk-sensitive distributional reinforcement learning with general function approximation (2024) — link
  62. Risk-sensitive policy optimization via predictive CVaR policy gradient (2024)
  63. Risk-sensitive Markov control processes (2013)
  64. Policy gradients for CVaR-constrained MDPs (2014)
  65. Risk-constrained reinforcement learning with percentile risk criteria (2018)
  66. Near-minimax-optimal risk-sensitive reinforcement learning with cvar (2023)
  67. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret (2020)
  68. Risk-sensitive reinforcement learning with function approximation: A debiasing approach (2021)
  69. Policy gradients with variance related risk criteria (2012) — link
  70. Actor-critic algorithms for risk-sensitive MDPs (2013)
  71. A block coordinate ascent algorithm for mean-variance optimization (2018)
  72. Provably efficient iterated cvar reinforcement learning with function approximation and human feedback (2023) — link
  73. Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path (2022) — link
  74. R-max-a general polynomial time algorithm for near-optimal reinforcement learning (2002)
  75. Distributionally robust optimization (2025)
  76. Fairness-oriented learning for optimal individualized treatment rules (2023)

2 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.