Paper: arXiv 2603.00738

Authors: Sebastien Lleo, Wolfgang Runggaldier

Abstract

This paper bridges reinforcement learning (RL) and risk-sensitive stochastic control by introducing a tractable exploration mechanism for policy search in risk-sensitive portfolio management, with known and unknown model parameters, that yields an endogenous relative-entropy regularization. We construct a discrete-time risk-sensitive benchmarked investment model. This model combines a factor-based asset universe with periodic portfolio rebalancing. Exploration is incorporated through user-specified Gaussian perturbations to baseline (exploitative) controls. The risk-sensitive stochastic control problem is solved analytically using the Free Energy-Entropy Duality. The Duality recasts the control problem as a linear-quadratic-Gaussian game and introduces a natural penalty for exploration. This approach yields simple sufficiency conditions for optimality. It also induces intuitive bounds on exploration based on risk sensitivity, asset covariance, and rebalancing frequency. Additionally, the optimal investment strategy can be interpreted through the lens of fractional Kelly strategies. By connecting risk-sensitive control theory and RL, this work provides a principled parametric family for policy-gradient implementations, guiding the design of RL methods.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 2.5/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper provides a rigorous mathematical framework connecting risk-sensitive stochastic control with reinforcement learning via the Free Energy-Entropy Duality. However, it is purely theoretical and lacks empirical backtesting, data validation, or publicly available code implementations.

Research Flowchart

  flowchart TD
    A[Goal: Integrate RL Exploration into Risk-Sensitive Portfolio Management] --> B[Inputs: Factor Assets, Risk Sensitivity, Rebalancing Frequency & Gaussian Perturbations]
    B --> C[Apply Free Energy-Entropy Duality]
    C --> D[Transform Problem into Linear-Quadratic-Gaussian Game with Endogenous Entropy Penalty]
    D --> E[Derive Analytical Optimal Strategy & Bounds on Exploration]
    E --> F[Outcomes: Fractional Kelly Interpretation & Principled Parametric Family for RL Policy Gradients]

Cited References (23)

  1. Exploratory Randomization for Discrete-Time Linear Exponential Quadratic Gaussian (LEQG) Problem — → 2501.06275
  2. Dual dominance: how Harry Markowitz and William Ziemba impacted portfolio management (2024) — link
  3. Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty — → 2404.12598
  4. On the separation of estimation and control in risk-sensitive investment problems under incomplete observation — → 2304.08910
  5. Recent advances in reinforcement learning in finance (2021) — link
  6. Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms (2021) — link
  7. Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach (2021) — link
  8. Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games (2021) — link
and 15 more
  1. Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon (2020) — link
  2. Continuous‐time mean–variance portfolio selection: A reinforcement learning framework (2019) — link
  3. Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator (2018) — link
  4. Risk-Sensitive Investment Management (2014) — link
  5. Risk-sensitive benchmarked asset management (2008) — link
  6. Risk-sensitive portfolio optimization on infinite time horizon (2002) — link
  7. Risk sensitive portfolio optimization (1999) — link
  8. Connections between stochastic control and dynamic games (1996) — link
  9. A Natural Actor-Critic Framework for Zero-Sum Markov Games (2022)
  10. Reinforcement Learning in Continuous Time and Space: A Stochastic Control Approach (2020)
  11. Reinforcement Learning, second edition: An Introduction (2018)
  12. Formulas for Discrete Time LQR, LQG, LEQG and Minimax LQG Optimal Control Problems (2008) — link
  13. Modelli Risolvibili per Problemi di Controllo di Sistemi Dinamici Imprecisi Mul-tivariati (1994)
  14. Risk-Sensitive Optimal Control (1990)
  15. (i) Duality–driven continuous-time shift: a piecewise-constant, right-continuous process ¯Γ = (¯ γ s ) s ∈ [0 ,T ] ∈ R d such that ¯ γ s is constant on each interval [ t k , t k +1 ), k = 0 , . .

3 of 23 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.