Paper: arXiv 2603.22880

Authors: Minkey Chang

Abstract

We study whether a risk-sensitive objective from asset-pricing theory – recursive utility – improves reinforcement learning for portfolio allocation. The Bellman equation under recursive utility involves a certainty equivalent (CE) of future value that has no closed form under observed returns; we approximate it by $K$-sample Monte Carlo and train actor-critic (PPO, A2C) on the resulting value target and an approximate advantage estimate (AAE) that generalizes the Bellman residual to multi-step with state-dependent weights. This formulation applies only to critic-based algorithms. On 10 chronological train/test splits of South Korean ETF data, the recursive-utility agent improves on the discounted (naive) baseline in Sharpe ratio, max drawdown, and cumulative return. Derivations, world model and metrics, and full result tables are in the appendices.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 6.0/10 (capped from 6.5: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a novel integration of economic recursive utility theory into modern actor-critic reinforcement learning algorithms (PPO, A2C) via approximate Bellman updates. The mathematical formulation is mathematically dense and well-derived, while the empirical evaluation on multi-period South Korean ETF data demonstrates meaningful performance improvements over baseline discounted approaches.

Research Flowchart

  flowchart TD
    A["Research Goal: Improve RL Portfolio Allocation<br/>using Asset-Pricing Recursive Utility"] --> B["Data Input: South Korean ETF Data<br/>(10 Chronological Train/Test Splits)"]
    
    B --> C["Core Methodology: Calculate Value Target<br/>via K-sample Monte Carlo Certainty Equivalent"]
    
    C --> D["Algorithm Training: Actor-Critic (PPO, A2C)<br/>with Approximate Advantage Estimate (AAE)"]
    
    D --> E["Evaluation vs. Naive Discounted Baseline"]
    
    E --> F["Outcomes: Recursive Utility Agent Achieves<br/>Higher Sharpe Ratio, Lower Max Drawdown & Better Cumulative Return"]

Cited References (66)

  1. Efficiently Modeling Long Sequences with Structured State Spaces (2022) — link
  2. SpaceTime: A Unified Framework for Learning in Time Series and Beyond (2023) — link
  3. FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance (2020)
  4. Macroeconomics and Reality (1980)
  5. Co-integration and Error Correction: Representation, Estimation, and Testing (1987)
  6. Statistical Analysis of Cointegration Vectors (1988)
  7. A New Approach to Linear Filtering and Prediction Problems (1960)
  8. Distribution of the Estimators for Autoregressive Time Series with a Unit Root (1979)
and 58 more
  1. Testing the Null Hypothesis of Stationarity against the Alternative of a Unit Root: How Sure Are We That Economic Time Series Have a Unit Root? (1992)
  2. Dynamic Factor Models (2011)
  3. Macroeconomic Forecasting Using Diffusion Indexes (2002)
  4. Estimation and Inference of Impulse Responses by Local Projections (2005)
  5. Local Projections and VARs Estimate the Same Impulse Responses (2021)
  6. What are the Effects of Monetary Policy on Output? Results from an Agnostic Identification Procedure (2005)
  7. Disentangling the Channels of the 2007-2009 Recession (2012)
  8. The Dynamic Effects of Aggregate Demand and Supply Disturbances (1989)
  9. Measuring the Effects of Monetary Policy: A Factor-Augmented Vector Autoregressive (FAVAR) Approach (2005)
  10. Time Varying Structural Vector Autoregressions and Monetary Policy (2005)
  11. Drifts and Volatilities: Monetary Policies and Outcomes in the Post WWII U.S. (2005)
  12. Forecasting with Bayesian Vector Autoregressions: Five Years of Experience (1986)
  13. Large Bayesian Vector Auto Regressions (2010)
  14. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) — link
  15. Small-Sample Confidence Intervals for Impulse Response Functions (1998)
  16. Structural Vector Autoregressive Analysis (2017)
  17. New Introduction to Multiple Time Series Analysis (2005)
  18. Time Series Analysis (1994)
  19. Time Series Analysis by State Space Methods (2012)
  20. Linear Systems (1980)
  21. Matrix Analysis (2012)
  22. Iterative Methods for Sparse Linear Systems (2003)
  23. An Algorithm for the Machine Calculation of Complex Fourier Series (1965)
  24. Identification in Parametric Models (1971)
  25. Comparing Predictive Accuracy (1995)
  26. Bootstrapping Autoregressions with Conditional Heteroskedasticity of Unknown Form (2004)
  27. System Identification: Theory for the User (1999)
  28. Simplified State Space Layers for Sequence Modeling (2023) — link
  29. HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020)
  30. Diagonal State Spaces are as Effective as Structured State Spaces (2022)
  31. Large Bayesian Vector Autoregressions (2019)
  32. Sparse Vector Autoregressive Modeling (2017)
  33. Macroeconomic Shocks and Their Propagation (2016)
  34. RetNet: Retention Networks (2023) — link
  35. Prior Selection for Vector Autoregressions (2015)
  36. Machine Learning Time Series Regressions with an Application to Nowcasting (2022)
  37. Large Bayesian Vector Autoregressions with Stochastic Volatility and Non-Conjugate Priors (2022)
  38. Portfolio Selection (1952)
  39. Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk (1964)
  40. Substitution, Risk Aversion, and the Temporal Behavior of Consumption and Asset Returns: A Theoretical Framework (1989)
  41. Exotic Preferences for Macroeconomists (2004)
  42. Optimum Consumption and Portfolio Rules in a Continuous-Time Model (1971)
  43. Reinforcement Learning: An Introduction (2018)
  44. Foundations of Deep Reinforcement Learning: Theory and Practice in Python (2020)
  45. Proximal Policy Optimization Algorithms (2017) — link
  46. Asynchronous Methods for Deep Reinforcement Learning (2016)
  47. High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015) — link
  48. Strategic Asset Allocation: Portfolio Choice for Long-Term Investors (2002)
  49. Dynamic Programming: Finite States (2025)
  50. Understanding Deep Learning (2024)
  51. Optimal Consumption and Portfolio Selection with Stochastic Differential Utility (1999)
  52. Portfolio Selection in Stochastic Environments (2007)
  53. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
  54. A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising (2020) — link
  55. An Adaptive Portfolio Trading System: A Risk-Return Trade-Off (2017)
  56. Time-Consistent Reinforcement Learning for Optimal Consumption under Epstein–Zin Preferences (2023) — link
  57. Nowcasting: The Real-Time Informational Content of Macroeconomic Data (2008)
  58. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (2021)

0 of 66 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.