Paper: arXiv 2311.06896

Authors: Nicole Bäuerle, Anna Jaśkiewicz

Abstract

The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term ‘risk-sensitive’ refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure and Conditional Value-at-Risk. We restrict our considerations to stationary problems with an infinite time horizon. Conditions are given under which optimal policies exist and solution procedures are explained. We present both the theory when the Optimized Certainty Equivalent is applied recursively as well as the case where it is applied to the cumulated reward. Discounted as well as non-discounted models are reviewed

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 1.5/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper provides a rigorous mathematical overview of risk-sensitive Markov Decision Processes using the Optimized Certainty Equivalent (including entropic risk and CVaR). However, as a purely theoretical survey covering existence conditions and solution procedures, it lacks empirical backtests or real-world dataset implementations.

Research Flowchart

  flowchart TD
    A[Research Goal: Review Risk-Sensitive Infinite-Horizon MDPs] --> B[Input Criteria & Framework]
    
    subgraph B [Input Criteria & Framework]
        B1[Optimized Certainty Equivalent: Entropic Risk & CVaR]
        B2[Discounted & Non-Discounted Models]
    End

    B --> C[Methodology: Compare Evaluators]
    
    C --> D[Recursive OCE Application]
    C --> E[Cumulated Reward OCE Application]
    
    D --> F[Computational Analysis: Bellman Operators & Policy Iteration]
    E --> F
    
    F --> G[Key Outcomes: Existence Conditions & Optimal Policy Algorithms]

Cited References (194)

  1. Real Analysis (1988)
  2. Connections between stochastic control and dynamic games (1996)
  3. Markov D ecision P rocesses with A pplications to F inance (2011)
  4. More risk-sensitive M arkov decision processes (2014)
  5. Partially observable risk-sensitive stopping problems in discrete time (2015)
  6. Large Deviations Techniques and Applications (1998)
  7. Portfolio selection (1952)
  8. Risk aversion and precautionary savings in dynamic settings (2019)
and 186 more
  1. Foundations of N on- S tationary D ynamic P rogramming with D iscrete Time Parameter (1970)
  2. Stationary policies in dynamic programming models under compactness assumptions (1983)
  3. Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal (1975)
  4. Risk-sensitive optimal control of hidden M arkov models: Structural results (1997)
  5. Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systems (1994)
  6. Risk sensitive M arkov decision processes (1997)
  7. Robust and risk-sensitive output feedback control for finite state machines and hidden M arkov models (1997)
  8. Risk-sensitive and minimax control of discrete-time, finite-state M arkov decision processes (1999)
  9. Nonexpected utility in macroeconomics (1990)
  10. Precautionary savings and the permanent income hypothesis (1993)
  11. Tutorial on risk neutral, distributionally robust and risk averse multistage stochastic programming (2021)
  12. Classical risk-averse control for a finite-horizon B orel model (2021)
  13. On optimizing the conditional value-at-risk of a maximum cost for risk-averse safety analysis (2023)
  14. Discounted linear exponential quadratic G aussian control (1995)
  15. Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games (1973)
  16. Risk-sensitive M arkov control processes (2013)
  17. A unified framework for risk-sensitive M arkov control processes (2014)
  18. Sequential decision making with coherent risk (2016)
  19. Incorporating convex risk measures into multistage stochastic programming algorithms (2022)
  20. Multistage stochastic programs with the entropic risk measure (2020)
  21. Evaluating policies in risk-averse multi-stage stochastic programming (2015)
  22. Convergence analysis of sampling-based decomposition methods for risk-averse multistage stochastic convex programs (2016)
  23. Robust optimal control using conditional risk mappings in infinite horizon (2018)
  24. Controlled M arkov decision processes with A V a R criteria for unbounded costs (2017)
  25. Markov decision processes with average-value-at-risk criteria (2011)
  26. Finite-horizon dynamic optimisation when the terminal reward is a concave functional of the distribution of the final state (1998)
  27. A recursive algorithm for multivariate risk measures and a set-valued B ellman’s principle (2017)
  28. Regret Bounds for M arkov Decision Processes with Recursive Optimized Certainty Equivalents (2023) — link
  29. Exponential B ellman equation and improved regret bounds for risk-sensitive reinforcement learning (2021)
  30. Entropic risk optimization in discounted M D P s (2023)
  31. Tight approximations of dynamic risk measures (2015)
  32. Reinforcement learning with dynamic convex risk measures (2023)
  33. On the convergence and optimality of policy gradient for M arkov coherent risk (2021) — link
  34. Contractive approximations in average M arkov decision chains driven by a risk-seeking controller (2023)
  35. Risk-Sensitive M arkov Decision Processes with Long-Run C V a R Criterion (2022) — link
  36. Ergodic risk-sensitive control of M arkov processes on countable state space revisited (2022)
  37. Average optimality for risk-sensitive control with general state space (2007)
  38. Ergodic risk-sensitive control— A survey (2023)
  39. An optimality system for finite average M arkov decision chains under risk-aversion (2012)
  40. A characterization of the optimal certainty equivalent of the average cost via the Arrow-Pratt sensitivity function (2016)
  41. A variational formula for risk-sensitive reward (2017)
  42. Linear and dynamic programs for risk-sensitive cost minimization (2021)
  43. Certainty equivalent control of discrete time M arkov processes with the average reward functional (2023)
  44. A learning algorithm for risk-sensitive cost (2008)
  45. Risk-Sensitive Control and an Abstract C ollatz– W ielandt Formula (2016)
  46. Value iteration for controlled M arkov chains with risk sensitive cost criterion (1999)
  47. Risk-sensitive average optimality for discrete-time M arkov decision processes (2023)
  48. A sensitivity formula for risk-sensitive cost and the actor–critic algorithm (2001)
  49. Q-learning for risk-sensitive control (2002)
  50. Linear and dynamic programming approaches to degenerate risk-sensitive reward processes (2017)
  51. Risk-sensitive optimal control for M arkov decision processes with monotone cost (2002)
  52. Controlled M arkov chains with risk-sensitive criteria: some (counter) examples (1998)
  53. Solution to the risk-sensitive average cost optimality equation in a class of M arkov decision processes with finite state space (2003)
  54. Solutions of the average cost optimality equation for finite M arkov decision chains: risk-sensitive and risk-neutral criteria (2009)
  55. Optimality equations and inequalities in a class of risk-sensitive average cost Markov decision chains (2010)
  56. Characterization of the optimal risk-sensitive average cost in denumerable M arkov decision chains (2018)
  57. Discounted approximations to the risk-sensitive average cost in finite M arkov chains (2017)
  58. Solution to the risk-sensitive average optimality equation in communicating M arkov decision chains with finite state space: A n alternative approach (2002)
  59. The vanishing discount approach in M arkov chains with risk-sensitive criteria (2000)
  60. A characterization of the optimal risk-sensitive average cost in finite controlled M arkov chains (2005)
  61. Necessary and sufficient conditions for a solution to the risk-sensitive P oisson equation on a finite state space (2009)
  62. Discounted approximations for risk-sensitive average criteria in M arkov decision chains with finite state space (2011)
  63. The value iteration algorithm in risk-sensitive average M arkov decision chains with finite state space (2003)
  64. Nonstationary value iteration in controlled M arkov chains with risk-sensitive average criterion (2005)
  65. The discounted method and equivalence of average criteria for risk-sensitive M arkov decision processes on Borel spaces (2010)
  66. Risk-sensitive average optimality for discrete-time M arkov decision processes (2023)
  67. Risk-sensitive control of discrete-time M arkov processes with infinite horizon (1999)
  68. Infinite horizon risk sensitive control of discrete time M arkov processes with small risk (2000)
  69. Infinite horizon risk sensitive control of discrete time M arkov processes under minorization property (2007)
  70. Risk-sensitive portfolio optimization with completely and partially observed factors (2004)
  71. Partially observable risk-sensitive M arkov decision processes (2017)
  72. Risk-sensitive control of finite state machines on an infinite horizon I (1997)
  73. Risk-sensitive control of finite state machines on an infinite horizon II (1999)
  74. Existence of risk-sensitive optimal stationary policies for controlled M arkov processes (1999)
  75. Risk sensitive control of M arkov processes in countable state space (1996)
  76. Inventory Control with an Exponential Utility Criterion (1992)
  77. Risk-sensitive M arkov decision processes (1972)
  78. A note on risk-sensitive control of invariant models (2007)
  79. A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost M D P (2022) — link
  80. Multiplicative M arkov decision chains (1984)
  81. Risk-sensitive linear/quadratic/ G aussian control (1981)
  82. Risk-Sensitive Optimal Control (1990)
  83. Risk-Sensitive M arkov Decision Processes with Combined Metrics of Mean and Variance (2020)
  84. Growth rates AND AVERAGE OPTIMALITY IN RISK–SENSITIVE M ARKOV DECISION CHAINS (2008)
  85. RISK-SENSITIVE AVERAGE OPTIMALITY IN M ARKOV DECISION PROCESSES (2018)
  86. Substitution, risk aversion and the temporal behavior of consumption and asset returns: A theoretical framework (1989)
  87. Risk-sensitive control of M arkov decision processes: A moment-based approach with target distributions (2020)
  88. Decision problems with expected utility critera, I : upper and lower convergent utility (1977)
  89. Decision problems with expected utility criteria, I I : stationarity (1977)
  90. Temporal Resolution of Uncertainty and Dynamic Choice Theory (1978)
  91. Stochastic optimization of forward recursive functions (2004)
  92. Discounted M D P ’s: Distribution functions and exponential utility maximization (1987)
  93. Markov decision processes with recursive risk measures (2022)
  94. Distributionally robust M arkov decision processes and their connection to risk measures (2022)
  95. Markov decision processes with iterated coherent risk measures (2014)
  96. A note on negative dynamic programming for risk-sensitive control (2008)
  97. Risk-averse control of undicounted transient M arkov models (2014)
  98. Discrete-time risk-aware optimal switching with non-adapted costs (2022)
  99. On recursive utilities with non-affine aggregator and conditional certainty equivalent (2020)
  100. A turnpike theorem for a risk-sensitive M arkov decision process with stopping (2006)
  101. A utility criterion for M arkov decision processes (1976)
  102. Risk-sensitive and robust decision-making: a C V a R optimization approach (2015)
  103. Discounted M arkov decision processes with utility constraints (2006)
  104. Time consistency and risk averse dynamic decision models: Definition, interpretation and practical consequences (2014)
  105. Risk-averse multi-stage stochastic optimization for surveillance and operations planning of a forest insect infestation (2022)
  106. A multi-product risk-averse newsvendor with exponential utility function (2011)
  107. Coherent risk measures in inventory problems (2007)
  108. Decomposition algorithms for risk-averse multistage stochastic programs with application to water allocation under uncertainty (2016)
  109. Regime switching optimal growth model with risk sensitive preferences (2022)
  110. Automating Vehicles by Risk-Averse Preview-based Q -Learning Algorithm (2022)
  111. Discrete-time risk sensitive portfolio optimization with proportional transaction costs (2023)
  112. Discrete time risk sensitive portfolio optimization with consumption and proportional transaction costs (2005)
  113. Risk-sensitive consumption and savings under rational inattention (2010)
  114. Risk-sensitive planning with probabilistic decision graphs (1994)
  115. Risk sensitive portfolio optimization (1999)
  116. The dynamics of risk-sensitive allocations (2005)
  117. The dynamics of risk-sensitive allocations (2015)
  118. Time-consistent, risk-averse dynamic pricing (2019)
  119. A survey on risk-averse and robust revenue management (2017)
  120. Risk-sensitivity in sensorimotor control (2011)
  121. Risk-sensitive optimal feedback control for haptic assistance (2012)
  122. Risk-sensitive optimal control of quantum systems (2004)
  123. Risk-sensitive control under M arkov modulated denial-of-service ( D o S ) attack strategies (2015)
  124. A M arkov decision model for a surveillance application and risk-sensitive M arkov decision processes (2010)
  125. Robust and risk-sensitive M arkov decision processes with applications to dynamic optimal reinsurance (2020)
  126. Robust, risk-sensitive, and data-driven control of M arkov decision processes (2007)
  127. Risk-sensitive and risk-neutral multiarmed bandits (2007)
  128. Stochastic optimal growth model with risk sensitive preferences (2018)
  129. A stochastic dynamic pricing and advertising model under risk aversion (2015)
  130. Stochastic dynamic multi-product pricing with dynamic advertising and adoption effects (2016)
  131. A generalized multi-period mean–variance portfolio optimization with M arkov switching parameters (2008)
  132. Dynamic mean-risk optimization in a binomial model (2009)
  133. Optimizing C onditional V alue-at- R isk in dynamic pricing (2018)
  134. Risk neutral and risk averse stochastic dual dynamic programming method (2013)
  135. Application of C V a R risk aversion approach in the expansion and operation planning and for setting the spot price in the B razilian hydrothermal interconnected system (2015)
  136. Practicality of nested risk measures for dynamic electric vehicle charging (2016) — link
  137. Risk-sensitive capacity control in revenue management (2007)
  138. Risk-sensitive dividend problems (2015)
  139. Optimal dividend payout model with risk sensitive preferences (2017)
  140. Risk sensitive control of finite state M arkov chains in discrete time, with applications to portfolio management (1999)
  141. Nested C onditional V alue-at- R isk portfolio selection: A model with temporal dependence driven by market-index volatility (2020)
  142. Optimal bidding of a virtual power plant on the Spanish day-ahead and intraday market for electricity (2020)
  143. Sequential Optimization of C V a R (2022) — link
  144. Structure of risk-averse multistage stochastic programs (2015)
  145. An old-new concept of convex risk measures: The optimized certainty equivalent (2007)
  146. Dynamic Programming, Vol. I: Foundations (2023)
  147. Economic Dynamics in Discrete Time (2020)
  148. Risk aversion in multistage stochastic programming: A modeling and algorithmic perspective (2016)
  149. A note on a new class of recursive utilities in M arkov decision processes (2017)
  150. Risk-averse dynamic programming for M arkov decision processes (2010)
  151. Measuring risk for income streams (2005)
  152. A value-of-information approach to measuring risk in multi-period economic activity (2006)
  153. Time-inconsistent multistage stochastic programs: Martingale bounds (2016)
  154. On solving multistage stochastic programs with coherent risk measures (2013)
  155. Minimax and risk averse multistage stochastic programming (2012)
  156. Convex and coherent risk measures (2010)
  157. Iterated risk measures for risk-sensitive M arkov decision processes with discounted cost (2011)
  158. Risk-sensitive real business cycles (2000)
  159. Dynamic programming for non-additive stochastic objectives (1996)
  160. Unique solutions for stochastic recursive utilities (2010)
  161. Convex dynamic programming with (bounded) recursive utility (2018)
  162. Do not blame B ellman: It is K oopmans’ fault (2021)
  163. Conditional decision processes with recursive function (1999)
  164. Solutions for some dynamic problems with uncertainty aversion (2001)
  165. Dynamic programming with recursive preferences: Optimality and applications (2018) — link
  166. Discrete-Time Markov Control Processes, Basic Optimality Criteria (1996)
  167. Examples in Markov Decision Processes (2013)
  168. Robust M arkov control processes (2014)
  169. Recursive smooth ambiguity preferences (2009)
  170. Markov decision processes under ambiguity (2020)
  171. Robustness and risk-sensitivity in Markov decision processes (2012)
  172. Robust Markov decision processes (2013)
  173. Distributionally robust Markov decision processes (2010)
  174. Robust control of Markov decision processes with uncertain transition matrices (2005)
  175. Robust dynamic programming (2005)
  176. Robust Markov control processes (2014)
  177. Ambiguity, learning, and asset returns (2012)
  178. Temporal resolution of uncertainty and recursive models of ambiguity aversion (2013)
  179. Reinforcement Learning: An Introduction (2018)
  180. Reinforcement Learning and Stochastic Optimization: A unified framework for sequential decisions (2022)
  181. Recent advances in reinforcement learning in finance (2023)
  182. Exposition of a new theory on the measurement of risk (1954)
  183. Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition) (2007)
  184. The theory of risk aversion (1971)
  185. Risk aversion in the small and in the large (1964)
  186. Markov Decision Processes: Discrete Stochastic Dynamic Programming (2014)

0 of 194 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.