Paper: arXiv 2501.16659

Authors: Yuling Max Chen, Bin Li, David Saunders

Abstract

Considering the continuous-time Mean-Variance (MV) portfolio optimization problem, we study a regime-switching market setting and apply reinforcement learning (RL) techniques to assist informed exploration within the control space. We introduce and solve the Exploratory Mean Variance with Regime Switching (EMVRS) problem. We also present a Policy Improvement Theorem. Further, we recognize that the widely applied Temporal Difference (TD) learning is not adequate for the EMVRS context, hence we consider Orthogonality Condition (OC) learning, leveraging the martingale property of the induced optimal value function from the analytical solution to EMVRS. We design a RL algorithm that has more meaningful parameterization using the market parameters and propose an updating scheme for each parameter. Our empirical results demonstrate the superiority of OC learning over TD learning with a clear convergence of the market parameters towards their corresponding ``grounding true" values in a simulated market scenario. In a real market data study, EMVRS with OC learning outperforms its counterparts with the highest mean and reasonably low volatility of the annualized portfolio returns.

Complexity vs Empirical Score

  • Math Complexity: 9.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; reported Sharpe ratio of 5.9269 is implausibly high for daily or lower-frequency data; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 7.0)

Why this score: The paper uses advanced continuous-time stochastic control, Lagrangian duality, and RL theory, indicating very high math complexity. It includes both simulated convergence tests and real-market backtests with performance metrics, showing strong empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: How to solve MV Portfolio Optimization<br>with Regime Switching (EMVRS) using RL?"] --> B["Methodology: Exploratory MV with Regime Switching<br>Policy Improvement Theorem"]
  B --> C["Key Innovation:<br>Orthogonality Condition (OC) Learning<br>vs Standard TD Learning"]
  C --> D["Data & Simulation:<br>Simulated Markets with 'True' Parameters"]
  D --> E["Computational Process:<br>RL Algorithm with Market Parameter Updates"]
  E --> F["Key Findings:<br>1. OC Learning converges to true parameters<br>2. OC outperforms TD in accuracy<br>3. Real market: Highest returns & low volatility"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes the Exploratory Mean‑Variance with Regime Switching (EMVRS) problem and an Orthogonality Condition (OC) learning algorithm for reinforcement learning in this setting.

Source sentence
We introduce and solve the Exploratory Mean Variance with Regime Switching (EMVRS) problem.

Assumes 3

  • There are only two market regimes (l = 2). model
    Source sentence
    For ease of presentation, we hereafter suppose that there are only two regimes, i.e., l = 2
  • Market parameters (volatility, correlation, interest rate) are constant within each regime over time. market
    Source sentence
    We further assume that the market parameters and the interest rates are constant in the same regime, regardless of the time.
  • The optimal policy distribution is Gaussian with entropy regularization (exploration). model
    Source sentence
    the RL framework offers an informed guidance of exploration within the control space, which is a Gaussian exploration with entropy regularization.

Data

Backtest sample: S&P 500 · equities · monthly

  • S&P500 daily index · Yahoo Finance market-prices public
    Source sentence
    we collect from Yahoo Finance daily data of the S&P500 (ticker: “ˆGSPC”) market index as the market portfolio
  • 3‑month US Treasury Bill rate · Yahoo Finance macro public
    Source sentence
    benchmark the 3-month US Treasury Bill (3mTbill) rate (ticker: “ˆIRX”) as the riskfree interest rate
  • Simulated market data synthetic unknown
    Source sentence
    Our empirical results demonstrate the superiority of OC learning over TD learning with a clear convergence of the market parameters towards their corresponding “grounding true” values in a simulated market scenario.

Recognised sources: Yahoo Finance

Costs

Unclear - sources disagree The platform classifier and the paper-text extraction give different answers, so no value is asserted.

Platform classifier: Not modeled

Paper text extraction: Not applicable

Finds 3

Reported performance (EMVRS with OC learning): Sharpe 5.93 · annual return 12.2%

  • OC learning outperforms TD learning, achieving higher mean returns and lower volatility, leading to significantly higher Sharpe ratios on real data. performance
    Source sentence
    While both the mean and volatility of the portfolio returns are generally higher for EMVRS than those for EMV, the Sharpe Ratios of EMVRS are significantly higher than those of the EMV.
  • Parameters of the RL model converge to their true market values in simulation studies. theoretical
    Source sentence
    the parameters of our proposed RL model can converge to their corresponding true values, with randomly chosen starting point.
  • TD learning yields lower mean returns but inflated volatilities, resulting in the lowest Sharpe ratios among all settings. performance
    Source sentence
    Under TD learning, EMVRS achieves lower mean portfolio returns but significantly inflated volatilities, which yields the lowest Sharpe Ratios under all investment settings.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 34 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (88)

  1. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  2. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  3. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  4. The reinforcement learning kelly strategy (2022)
  5. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022) — link
  6. Asset and liability management under a continuous-time mean–variance optimization framework (2006)
  7. Continuous-time portfolio selection with liability: Mean–variance model and stochastic LQ approach (2008)
  8. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation (2000)
and 80 more
  1. q-Learning in continuous time (2023)
  2. Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching (2024) — link
  3. Learning equilibrium mean-variance strategy (2023)
  4. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  5. Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market (2024)
  6. Portfolio selection (1952)
  7. Adaptive Control Processes-A Guided Tour (1961)
  8. Dynamic programming (1966)
  9. A stochastic reinforcement learning algorithm for learning real-valued functions (1990)
  10. Pattern-recognizing stochastic learning automata (1985)
  11. I. On governors (1868)
  12. Q-learning (1992)
  13. Policy gradient methods for reinforcement learning with function approximation (1999)
  14. PORTFOLIO SELECTION* (1952) — link
  15. Stochastic adaptive control methods: a survey (1975)
  16. A survey of some results in stochastic adaptive control (1985)
  17. A survey of numerical methods for optimal control (2009)
  18. Recent developments in machine learning methods for stochastic control and games (2023) — link
  19. Portfolio selection problems with markowitz’s mean–variance framework: a review of literature (2018)
  20. A comprehensive review of deterministic models and applications for mean-variance portfolio optimization (2019)
  21. Reinforcement learning: A survey (1996)
  22. Learning to predict by the methods of temporal differences (1988)
  23. Deep reinforcement learning: A brief survey (2017)
  24. Model-free reinforcement learning for financial portfolios: a brief survey (2019) — link
  25. Risk-Averse Bayes-Adaptive Reinforcement Learning (2021)
  26. Efficient Risk-Averse Reinforcement Learning (2022)
  27. Risk-Averse Reinforcement Learning for Portfolio Optimization (2021)
  28. Risk-Averse Distributional Reinforcement Learning: A CVaR Optimization Approach (2019) — link
  29. Mean-variance policy iteration for risk-averse reinforcement learning (2021)
  30. Reinforcement Learning with Non-Exponential Discounting (2022)
  31. Delayed Geometric Discounts: An Alternative Criterion for Reinforcement Learning (2022) — link
  32. UGAE : A Novel Approach to Non-exponential Discounting (2023) — link
  33. Average reward adjusted discounted reinforcement learning: Near-blackwell-optimal policies for real-world applications (2020) — link
  34. A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems (2021) — link
  35. Time-inconsistent objectives in reinforcement learning (2021)
  36. Reinventing Policy Iteration under Time Inconsistency (2022) — link
  37. Markov Decision Processes with Time-Varying Geometric Discounting (2023)
  38. Markowitz’s mean-variance portfolio selection with regime switching: A continuous-time model (2003)
  39. Markowitz’s mean-variance portfolio selection with regime switching: From discrete-time models to their continuous-time limits (2004)
  40. Markowitz’s mean-variance asset–liability management with regime switching: A multi-period model (2011)
  41. Markowitz’s mean-variance asset-liability management with regime switching: A continuous-time model (2008)
  42. Multi-period mean-variance portfolio selection with markov regime switching and uncertain time-horizon (2011)
  43. Multi-period markowitz’s mean–variance portfolio selection with state-dependent exit probability (2014)
  44. Explicit solutions of consumption-investment problems in financial markets with regime switching (2009)
  45. A stochastic control problem with regime switching (2021)
  46. Asset allocation under stochastic interest rate with regime switching (2012)
  47. Classical and singular stochastic control for the optimal dividend policy when there is regime switching (2011)
  48. Stochastic optimal control for nonlinear Markov jump diffusion processes (2012)
  49. Identifying bull and bear markets in stock returns (2000)
  50. International asset allocation with regime shifts (2002)
  51. How regimes affect asset allocation (2004)
  52. Dynamic asset allocation for varied financial markets under regime switching framework (2014)
  53. Neural Ordinary Differential Equations (2018)
  54. Attention Is All You Need (2017) — link
  55. A hidden markov model variant for sequence classification (2011)
  56. Similarity-based classification of sequences using hidden markov models (2004)
  57. Improving time series classification using hidden markov models (2012)
  58. The viterbi algorithm (1973)
  59. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm (1967) — link
  60. Partially observable markov decision processes (2012)
  61. Continuous-time stochastic control and optimization with financial applications (2009)
  62. Stochastic control and application to finance (2002)
  63. Machine learning in Finance (2020)
  64. Dynamic programming and optimal control: Volume I (2012)
  65. Dynamic programming and optimal control 4th edition, volume ii (2015)
  66. Partial differential equations with numerical methods (2003)
  67. The behavior of organisms: An experimental analysis (1938)
  68. Investment Science (2013)
  69. The Econometrics of Financial Markets (1997)
  70. A general theory of Markovian time inconsistent stochastic control problems (2010)
  71. A theory of Markovian time-inconsistent stochastic control in discrete time (2014)
  72. A theory of markovian time inconsistent stochastic control in continuous time (2016)
  73. Time-Inconsistent Control Theory with Finance Applications (2021)
  74. PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019) — link
  75. Utility Theory for Decision Making (1979) — link
  76. Anomalies: Utility maximization and experienced utility (2006)
  77. Anomalies: preference reversals (1990)
  78. Playing Atari with Deep Reinforcement Learning (2013) — link
  79. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  80. Reinforcement learning for options trading (2021)

0 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.