Paper: arXiv 2604.22188

Authors: Thai Nguyen, Pertiny Nkuize

Abstract

We study the problem of optimal portfolio selection under stochastic volatility within a continuous time reinforcement learning framework with portfolio constraints. Exploration is modeled through entropy-regularized relaxed controls, where the investor selects probability distributions over admissible portfolio allocations rather than deterministic strategies. Using dynamic programming arguments, we derive the associated entropy-regularized Hamilton-Jacobi-Bellman equation, whose Hamiltonian involves optimization over probability measures supported on a compact control set. We show that the optimal exploratory policy takes the form of a truncated Gaussian distribution characterized by spatial derivatives of the solution of the resulting nonlinear quasilinear parabolic partial differential equation. Under suitable structural conditions on the model coefficients, we prove the existence of classical solutions to this nonlinear HJB equation for the value function. We then establish a verification theorem and analyze the policy-improvement structure induced by the entropy-regularized Hamiltonian, showing how the resulting sequence of PDEs provides a continuous-time interpretation of actor-critic learning dynamics. Finally, our PDE analysis with a semi-closed form of optimal value and optimal policy enables the design of an implementable reinforcement learning algorithm by recasting the optimal problem in a martingale framework.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 2.5/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper presents a rigorous theoretical framework using nonlinear parabolic PDEs, entropy regularization, and verification theorems for continuous-time RL under stochastic volatility. However, it is purely theoretical with no empirical backtesting or real-world dataset validation included in the provided abstract.

Research Flowchart

  flowchart TD
    A[Goal: Optimal Portfolio Selection under Stochastic Volatility via Continuous-Time RL] --> B[Model Inputs: Portfolio Constraints, Stochastic Volatility & Entropy-Regularized Controls]
    B --> C[Derive Entropy-Regularized HJB Equation via Dynamic Programming]
    C --> D[Prove Existence of Classical Solutions & Formulate Verification Theorem]
    D --> E[Establish Policy Improvement & Actor-Critic Dynamics via PDE Sequence]
    E --> F[Outcomes: Truncated Gaussian Exploratory Policy & Martingale-Based RL Algorithm]

Cited References (73)

  1. Lifetime portfolio selection under uncertainty: The continuous-time case (1969)
  2. Optimum consumption and portfolio rules in a continuous-time model (1975)
  3. Optimal portfolio and consumption decisions for a ``small investor’’ on a finite horizon (1987)
  4. Methods of mathematical finance (1998)
  5. Stochastic controls: H amiltonian systems and HJB equations (1999)
  6. Controlled M arkov processes and viscosity solutions (2006)
  7. Continuous-time stochastic control and optimization with financial applications (2009)
  8. Convex duality in constrained portfolio optimization (1992)
and 65 more
  1. Hedging contingent claims with constrained portfolios (1993)
  2. Optimal investment and consumption with transaction costs (1994)
  3. Optimal consumption from investment and random endowment in incomplete semimartingale markets (2003)
  4. A closed-form solution for options with stochastic volatility with applications to bond and currency options (1993)
  5. Optimal portfolios and H eston’s stochastic volatility model: an explicit solution for power utility (2005)
  6. Dynamic consumption and portfolio choice with stochastic volatility in incomplete markets (2005)
  7. Portfolio selection in stochastic environments (2007)
  8. On estimating the expected return on the market: An exploratory investigation (1980)
  9. Portfolio selection and asset pricing models (2000)
  10. Robust control and model uncertainty (2001)
  11. Robustness (2011)
  12. Robust portfolio rules and asset pricing (2004)
  13. Robust portfolio rules and detection-error probabilities for a mean-reverting risk premium (2006)
  14. Risk-sensitive dynamic asset management (1999)
  15. Reinforcement learning: A n introduction (1998)
  16. Reinforcement learning in continuous time and space (2000)
  17. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  18. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  19. Learning M erton’s strategies in an incomplete market: recursive entropy regularization and biased G aussian exploration (2023) — link
  20. Continuous-time optimal investment with portfolio constraints: A reinforcement learning approach (2026)
  21. Reinforcement learning for jump-diffusions, with financial applications (2024)
  22. Continuous time reinforcement learning: A random measure approach (2026)
  23. Linear and quasi-linear equations of parabolic type (1968)
  24. Nonlinear elliptic and parabolic equations of the second order (1987)
  25. Partial differential equations (2022)
  26. Numerical methods for stochastic control problems in continuous time (1990)
  27. Applications of variational inequalities in stochastic control (2011)
  28. Backward stochastic differential equations in finance (1997)
  29. Probabilistic interpretation for systems of quasilinear parabolic partial differential equations (1991)
  30. Backward stochastic differential equations and quasilinear parabolic partial differential equations (2005)
  31. A generalized dynamic programming principle and H amilton– J acobi– B ellman equation (1992)
  32. Stochastic differential utility (1992)
  33. Introduction au calcul stochastique appliqu 'e `a la finance (1997)
  34. Two-scale stochastic systems: A symptotic analysis and control (2003)
  35. Simulation of square-root processes made simple: applications to the H eston model (2024)
  36. Investment science (1998)
  37. Second order parabolic differential equations (1996)
  38. Policy gradient and actor-critic learning in continuous time and space: T heory and algorithms (2022)
  39. Viscosity solutions of H amilton– J acobi equations (1983)
  40. User’s guide to viscosity solutions of second order partial differential equations (1992)
  41. Optimal consumption and portfolio policies when asset prices follow a diffusion process (1989)
  42. Consumption and portfolio decisions when expected returns are time varying (1999)
  43. Strategic asset allocation: P ortfolio choice for long-term investors (2002)
  44. Dynamic nonmyopic portfolio behavior (1996)
  45. Portfolio optimization with M arkov-modulated stock prices and interest rates (2004)
  46. Continuous multivariate distributions (2002)
  47. Computational methods for option pricing (2005)
  48. Numerical methods for controlled H amilton– J acobi– B ellman PDE s in finance (2007)
  49. Large deviations for M arkov processes and the asymptotic evaluation of certain M arkov process expectations for large times (2006)
  50. Optimal consumption and equilibrium prices with portfolio constraints and stochastic income (1997)
  51. Jump-diffusions with controlled jumps: E xistence and numerical methods (2000)
  52. Actor-critic algorithms (1999)
  53. Natural actor–critic algorithms (2009)
  54. Policy gradient methods for reinforcement learning with function approximation (1999)
  55. Reinforcement learning: A n introduction (2018)
  56. Efficient computation of optimal actions (2009)
  57. Stochastic optimal control as approximate inference: A new perspective (2013)
  58. Applications to partial differential equations—nonlinear equations (1983)
  59. Estimation of parameters of doubly truncated normal distribution from first four sample moments (1966)
  60. Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market (2024)
  61. Machine learning and speed in high frequency trading (2022)
  62. Learning, information processing and order submission in limit order markets (2015)
  63. Reinforcement learning for household finance: D esigning policy via responsiveness (2026)
  64. Heterogeneous trading strategies with adaptive fuzzy actor–critic reinforcement learning: A behavioral approach (2010)
  65. Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach (2022)

0 of 73 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.