Paper: arXiv 2505.04553

Authors: Shanyu Han, Yang Liu, Xiang Yu

Abstract

We propose a reinforcement learning (RL) framework under a broad class of risk objectives, characterized by convex scoring functions. This class covers many common risk measures, such as variance, Expected Shortfall, entropic Value-at-Risk, and mean-risk utility. To resolve the time-inconsistency issue, we consider an augmented state space and an auxiliary variable and recast the problem as a two-state optimization problem. We propose a customized Actor-Critic algorithm and establish some theoretical approximation guarantees. A key theoretical contribution is that our results do not require the Markov decision process to be continuous. Additionally, we propose an auxiliary variable sampling method inspired by the alternating minimization algorithm, which is convergent under certain conditions. We validate our approach in simulation experiments with a financial application in statistical arbitrage trading, demonstrating the effectiveness of the algorithm.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 5.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a mathematically rigorous theoretical foundation for risk-sensitive RL using convex scoring functions and dynamic programming over augmented state spaces. While supported by theoretical proofs and simulation experiments in statistical arbitrage, empirical validation on extensive real-world datasets is relatively limited.

Research Flowchart

  flowchart TD
    A["Research Goal: Solve Risk-Sensitive RL for Convex Scoring Functions"] --> B["Problem Reformulation: Augment State Space & Add Auxiliary Variable"]
    
    subgraph Methodology & Computation
        B --> C["Customized Actor-Critic Algorithm"]
        D["Auxiliary Variable Sampling (Alternating Minimization)"] --> C
    end
    
    E["Data/Inputs: Financial Market Data & MDP (Discrete/Non-continuous)"] --> C
    
    C --> F["Theoretical Outcomes: Convergence Guarantees ohne MDP Continuity"]
    C --> G["Practical Outcomes: Effective Statistical Arbitrage Trading"]

Cited References (79)

  1. Robust Risk-Aware Reinforcement Learning (2022) — link
  2. A Comprehensive Survey on Safe Reinforcement Learning (2015) — link
  3. Dynamic Coherent Risk Measures (2004) — link
  4. Markov Decision Processes with Recursive Risk Measures (2022) — link
  5. Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents (2023) — link
  6. A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (2024) — link
  7. Reinforcement Learning with Dynamic Convex Risk Measures (2024) — link
  8. Risk-averse Dynamic Programming for Markov Decision Processes (2010) — link
and 71 more
  1. Risk-sensitive Markov Decision Problems under Model Uncertainty: Finite Time Horizon Case (2020) — link
  2. Risk-sensitive Markov Decision Process and Learning under General Utility Functions (2023) — link
  3. Markov Decision Processes with Risk-Sensitive Criteria: An Overview — → 2311.06896
  4. Reinforcement Learning in Economics and Finance (2021) — link
  5. Optimization of $\text C $onditional $\text Value-at-Risk $ (2000)
  6. End-to-end training of deep visuomotor policies (2016)
  7. Continuous control with deep reinforcement learning (2015) — link
  8. Mastering the game of go with deep neural networks and tree search (2016)
  9. Reinforcement learning and deep learning based lateral control for autonomous driving [application notes] (2019) — link
  10. Playing atari with deep reinforcement learning (2013) — link
  11. Recent advances in reinforcement learning in finance (2023) — link
  12. Hedging in incomplete markets with HARA utility (1997)
  13. Continuous-time mean-variance portfolio selection: A stochastic $\text LQ $ framework (2000)
  14. Portfolio choice via quantiles (2011)
  15. Precision medicine, $\text AI $, and the future of personalized health care (2021) — link
  16. The users’ perspective on the privacy-utility trade-offs in health recommender systems (2019) — link
  17. Multilayer feedforward networks are universal approximators (1989)
  18. A survey of actor-critic reinforcement learning: Standard and natural policy gradients (2012)
  19. Onactor-critic algorithms (2003)
  20. Policy gradient methods for reinforcement learning with function approximation (1999)
  21. Reinforcement Learning: An Introduction (2018)
  22. Natural actor-critic (2008) — link
  23. Quantitative risk management: Concepts, techniques and tools: Revised edition (2015)
  24. Why scoring functions cannot assess tail properties (2019) — link
  25. Stochastic Finance. An Introduction in Discrete Time (2016)
  26. A framework for measures of risk under uncertainty (2024)
  27. Convolution Bounds on Quantile Aggregation (2025)
  28. A risk measurement approach from risk-averse stochastic optimization of score functions (2025)
  29. Generalized deviations in risk analysis (2006) — link
  30. Coherence and elicitability (2016)
  31. Higher order elicitability and $\text Osband $’s principle (2016)
  32. On elicitable risk measures (2015)
  33. Bayes risk, elicitability, and the Expected Shortfall (2021)
  34. Regulatory arbitrage of risk measures (2016)
  35. Robust Elicitable Functionals — → 2409.04412
  36. On the theory of policy gradient methods: Optimality, approximation, and distribution shift (2021)
  37. Adaptive alternating minimization algorithms (2007)
  38. Information geometry and alternating minimization procedures (1984)
  39. Sequential decision making with coherent risk (2016)
  40. The fundamental risk quadrangle in risk management, optimization and statistical estimation (2013)
  41. Time-inconsistency of VaR and time-consistent alternatives (2009)
  42. Mean-variance policy iteration for risk-averse reinforcement learning (2021)
  43. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  44. Markov decision processes with average-value-at-risk criteria (2011)
  45. Policy gradient based entropic-var optimization in risk-sensitive reinforcement learning (2022)
  46. Markov decision processes with recursive risk measures (2022)
  47. A Markovian decision process (1957)
  48. Infinite-horizon policy-gradient estimation (2001)
  49. Risk-sensitive and robust decision-making: a $\text CVaR $ optimization approach (2015)
  50. Optimizing the $\text CVaR $ via sampling (2015)
  51. Risk-sensitive learning via minimization of empirical conditional value-at-risk (2007)
  52. Provably efficient risk-sensitive reinforcement learning: Iterated $\text CVaR $ and worst path (2022) — link
  53. ACReL: Adversarial Conditional value-at-risk Reinforcement Learning (2021) — link
  54. On elicitable risk measures (2015)
  55. Making and evaluating point forecasts (2011)
  56. Transformations of predictions and realizations in consistent scoring functions (2025) — link
  57. How does loss function affect generalization performance of deep learning? Application to human age estimation (2021)
  58. Elicitation complexity of statistical properties (2021)
  59. Elicitability and identifiability of tail risk measures — → 2404.14136
  60. Ordering and Inequalities of Mixtures on Risk Aggregation (2022)
  61. Sensitivity measures based on scoring functions (2023)
  62. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  63. Optimal mean-variance portfolio selection (2017)
  64. Mean–variance portfolio optimization with state-dependent risk aversion (2014)
  65. Entropic value-at-risk: A new coherent risk measure (2012)
  66. An old-new concept of convex risk measures: The optimized certainty equivalent (2007)
  67. Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty — → 2404.12598
  68. Risk-constrained reinforcement learning with percentile risk criteria (2018)
  69. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret (2020)
  70. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning (2021)
  71. Efficient risk-averse reinforcement learning (2022)

4 of 79 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.