Paper: arXiv 2412.10692

Authors: Huy Chau, Duy Nguyen, Thai Nguyen

Abstract

In a reinforcement learning (RL) framework, we study the exploratory version of the continuous time expected utility (EU) maximization problem with a portfolio constraint that includes widely-used financial regulations such as short-selling constraints and borrowing prohibition. The optimal feedback policy of the exploratory unconstrained classical EU problem is shown to be Gaussian. In the case where the portfolio weight is constrained to a given interval, the corresponding constrained optimal exploratory policy follows a truncated Gaussian distribution. We verify that the closed form optimal solution obtained for logarithmic utility and quadratic utility for both unconstrained and constrained situations converge to the non-exploratory expected utility counterpart when the exploration weight goes to zero. Finally, we establish a policy improvement theorem and devise an implementable reinforcement learning algorithm by casting the optimal problem in a martingale framework. Our numerical examples show that exploration leads to an optimal wealth process that is more dispersedly distributed with heavier tail compared to that of the case without exploration. This effect becomes less significant as the exploration parameter is smaller. Moreover, the numerical implementation also confirms the intuitive understanding that a broader domain of investment opportunities necessitates a higher exploration cost. Notably, when subjected to both short-selling and money borrowing constraints, the exploration cost becomes negligible compared to the unconstrained case.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper provides a rigorous continuous-time mathematical framework for exploratory stochastic optimal control under portfolio constraints, deriving closed-form truncated Gaussian policies. However, its empirical validation is limited to synthetic numerical simulations rather than real-market backtesting.

Research Flowchart

  flowchart TD
    A["Research Goal: Solve Continuous-Time Optimal Investment with Portfolio Constraints via RL"] --> B["Problem Setup & Framework: Continuous-Time Exploratory EU Maximization + Constraints"]
    B --> C["Theoretical Derivation: Obtain Optimal Exploratory Policies"]
    C --> D["Algorithm Design: Martingale-Based Policy Improvement RL Algorithm"]
    D --> E["Numerical Simulations: Test Log & Quadratic Utilities under Constraints"]
    
    C --> F1["Findings: Unconstrained = Gaussian Policy; Interval Constrained = Truncated Gaussian"]
    E --> F2["Findings: Convergence to Non-Exploratory EU as Exploration Weight -> 0"]
    E --> F3["Findings: Constraints Reduce Exploration Cost; Exploration Yields Heavy-Tailed Wealth"]

Cited References (130)

  1. Mean-variance hedging in continuous time (1991)
  2. Compactification methods in the control of degenerate diffusions: existence of an optimal control (1987)
  3. On stochastic relaxed control for partially observed diffusions (1984)
  4. Generalized solutions in optimal stochastic control (1976)
  5. Learning equilibrium mean-variance strategy (2023)
  6. A dynamic mean-variance analysis for log returns (2021)
  7. A robust optimization approach to inventory theory (2006)
  8. Reinforcement learning methods for operations research applications: The order release problem (2019)
and 122 more
  1. Reinforcement learning in economics and finance (2021)
  2. The strong case for the generalized logarithmic utility model as the premier model of financial markets (1977)
  3. Mean-variance approximations to expected logarithmic utility (1983)
  4. Investment science (1998)
  5. Mean-Variance Efficient Reinforcement Learning by Expected Quadratic Utility Maximization (2020) — link
  6. Continuous-time mean-variance portfolio selection with bankruptcy prohibition (2005)
  7. Dynamic mean-variance portfolio selection with no-shorting constraints (2002)
  8. Continuous-time Markowitz’s model with constraints on wealth and portfolio (2016)
  9. On the equivalence of quadratic optimization problems commonly used in portfolio theory (2013)
  10. Near-optimal asset allocation in financial markets with trading constraints (2022)
  11. Optimization of consumption with labor income (1998)
  12. Optimal portfolios for logarithmic utility (2000)
  13. Optimum consumption and portfolio rules in a continuous-time model (1975)
  14. On optimal constrained investment strategies for long-term savers in stochastic environments and probability hedging (2023)
  15. Optimal investment and consumption when allowing terminal debt (2017)
  16. Deep reinforcement learning for the optimal placement of cryptocurrency limit orders (2022)
  17. Dynamic selective maintenance optimization for multi-state systems over a finite horizon: A deep reinforcement learning approach (2020)
  18. Multi-agent reinforcement learning algorithm to solve a partially-observable multi-agent problem in disaster response (2021)
  19. Reinforcement learning with internal expectation for the random neural network (2000)
  20. Reinforcement learning for long-run average cost (2004)
  21. Choquet regularization for reinforcement learning (2022) — link
  22. On stochastic gradient langevin dynamics with dependent data streams: The fully nonconvex case (2021)
  23. Bayesian learning via stochastic gradient Langevin dynamics (2011)
  24. Optimal consumption and equilibrium prices with portfolio constraints and stochastic income (1997)
  25. Large deviations for Markov processes and the asymptotic evaluation of certain Markov process expectations for large times (2006)
  26. Continuous multivariate distributions, Volume 1: Models and applications (2004)
  27. I-divergence geometry of probability distributions and minimization problems (1975)
  28. Partial differential equations of parabolic type (2008)
  29. Convergence of policy improvement for entropy-regularized stochastic control problems (2022) — link
  30. Reinforcement learning and stochastic optimisation (2022)
  31. Exploratory HJB equations and their convergence (2022)
  32. Robust exploratory mean-variance problem with drift uncertainty (2021) — link
  33. A general framework for learning mean-field games (2022)
  34. Residual algorithms: Reinforcement learning with function approximation (1995)
  35. Temporal-difference methods and Markov models (1993)
  36. Policy iterations for reinforcement learning problems in continuous time and space—Fundamental theory and methods (2021)
  37. Reinforcement learning using a continuous time actor-critic framework with spiking neurons (2013)
  38. Reinforcement learning in continuous time and space (2000)
  39. Reinforcement learning: A survey (1996)
  40. Information theoretic MPC for model-based reinforcement learning (2017)
  41. Reinforcement learning: A tutorial survey and recent advances (2009)
  42. Reinforcement learning and optimal control (2019)
  43. Reinforcement learning for combinatorial optimization: A survey (2021)
  44. Mastering the game of go without human knowledge (2017)
  45. Mastering the game of Go with deep neural networks and tree search (2016)
  46. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  47. Learning equilibrium mean-variance strategy (2020)
  48. Large scale continuous-time mean-variance portfolio allocation via reinforcement learning (2019) — link
  49. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  50. Performance functions and reinforcement learning for trading systems and portfolios (1998)
  51. A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
  52. Reinforcement learning for optimized trade execution (2006)
  53. Recent advances in reinforcement learning in finance (2021) — link
  54. Reinforcement learning: An introduction (2018)
  55. Investment science (1998)
  56. Controlled Markov processes and viscosity solutions (2006)
  57. An elementary approach to the Merton problem (2021)
  58. Continuous-time stochastic control and optimization with financial applications (2009)
  59. Q -Learning in continuous time (2023)
  60. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  61. Deep learning (2016)
  62. Exploratory HJB equations and their convergence (2021) — link
  63. The econometrics of financial markets (1998)
  64. Reinforcement Learning in Continuous Time and Space: A Stochastic Control Approach. (2020)
  65. Mean-variance portfolio selection for partially observed point processes (2020)
  66. Exploratory LQG mean field games with entropy regularization (2022)
  67. Recent Developments in Machine Learning Methods for Stochastic Control and Games (2022)
  68. Methods of mathematical finance (1998)
  69. Modelling Financial Derivatives with Mathematica \textregistered (1998)
  70. Pricing continuous Asian options: a comparison of Monte Carlo and Laplace transform inversion methods (1999)
  71. Monte Carlo algorithms for default timing problems (2011)
  72. Modeling term structures of defaultable bonds (1999)
  73. A closed-form expansion approach for pricing discretely monitored variance swaps (2015)
  74. The Effect of Jumps and Discrete Sampling on Volatility and Variance Swaps (2008)
  75. A General Framework for Pricing Asian Options Under Markov Processes (2015)
  76. Pricing Asian options under a hyper-exponential jump diffusion model (2012)
  77. A semi-analytical method for pricing and hedging continuously sample arithmetic average rate options (2001)
  78. Pricing continuously sampled Asian option with perturbation method (2003)
  79. Theory of Financial Decision Making (1987)
  80. Estimating security price derivatives using simulation (1996)
  81. A pricing method for options based on average asset values (1990)
  82. Connecting discrete and continuous path-dependent options (1999)
  83. Robust numerical methods for PDE models of Asian options (1997)
  84. The Value of an Asian Option (1995)
  85. A jump diffusion model for option pricing (2002)
  86. A new PDE approach for pricing arithmetic average Asian options (2001)
  87. Numerical Inversion of Laplace Transforms: An Efficient Improvement to Dubner and Abate’s Method (1974)
  88. The integral of geometric Brownian motion (2001)
  89. Bessel processes and a functional of Brownian motion (2004)
  90. Bessel processes, stochastic volatility and timer options (2014)
  91. The volatility surface-a practitioner’s guide (2006)
  92. Mathematical methods for financial markets (2009)
  93. Introduction to stochastic calculus with applications, 2nd Edition (2005)
  94. Semi-stable Markov processes. I (1972)
  95. Distributions, processes and statistics on spheres (1997)
  96. The GARCH option pricing model (1995)
  97. The spectral representation of Bessel processes with constant drift: applications in queuing and finance (2004)
  98. Spectral Expansions for Asian (Average Price) Options (2004) — link
  99. A study of the Hartman-Watson distribution motivated by numerical problems related to the pricing of Asian options (2004)
  100. Martingale property and pricing for time-homogeneous diffusion models in finance (“2013” volume = “”)
  101. Implied volatility asymptotics (“2012” volume = “”)
  102. Utility based pricing and exercising of real options under geometric mean reversion and risk aversion toward idiosyncratic risk (2008)
  103. Bessel processes, Asian options, and perpetuities (1993)
  104. Bessel processes, the integral of geometric Brownian motion, and Asian options (2004)
  105. Density of Generalized Verhulst Process and Bessel Process with Constant Drift (2015)
  106. Pricing timer options (2011)
  107. Laplace transforms for integrals of Markov Processes (2005)
  108. The Laplace transform of hitting times of integrated geometric Brownian motion (2013)
  109. Handbook of Brownian Motion-Facts and Formulae, 2nd edition (2002)
  110. Brownian Motion and Stochastic Calculus, 2nd edition (1991)
  111. Explicit formulas for Laplace transforms of stochastic integrals (2008)
  112. Continuous martingales and Brownian motions, 3rd edition (1999)
  113. Limit theorems for stochastic processes, 2nd ed (2003)
  114. A new proof for the conditions of Novikov and Kazamaki (2013)
  115. Option Valuation under Stochastic Volatility with Mathematica Code (2000)
  116. On the martingale property of certain local martingales (2012)
  117. Separating times for measures on filtered spaces (2004)
  118. A new approach of option pricing under stochastic volatility (2007)
  119. On the absolute continuity and singularity of measures on filtered spaces: seperating times (2006)
  120. Special Functions and Their Applications (1965)
  121. On some exponential functionals of Brownian motion (1992)
  122. Exponential functionals of Brownian motion and related processes (2001)

0 of 130 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.