Paper: arXiv 2312.15385

Authors: Xiangyu Cui, Xun Li, Yun Shi, Si Zhao

Abstract

This paper studies a discrete-time mean-variance model based on reinforcement learning. Compared with its continuous-time counterpart in \cite{“zhou2020mv”}, the discrete-time model makes more general assumptions about the asset’s return distribution. Using entropy to measure the cost of exploration, we derive the optimal investment strategy, whose density function is also Gaussian type. Additionally, we design the corresponding reinforcement learning algorithm. Both simulation experiments and empirical analysis indicate that our discrete-time model exhibits better applicability when analyzing real-world data than the continuous-time model.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 6.0/10 (capped from 6.5: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 6.5)

Why this score: The paper employs dense stochastic control and reinforcement learning theory, including derivations of Gaussian optimal policies and convergence proofs (high math). It validates the model with both simulation experiments and an empirical backtest on S&P 500 data, demonstrating a focus on real-world applicability (moderate-to-high rigor).

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Discrete-Time Mean-Variance Model<br>Based on Reinforcement Learning"] --> B["Methodology: Derive Optimal Strategy"]
  B --> C["Methodology: Design RL Algorithm<br>with Entropy Regularization"]
  C --> D["Data: Simulation &<br>Empirical Market Data"]
  D --> E["Computation: Apply Model & Algorithm"]
  E --> F["Outcomes: Gaussian Type<br>Optimal Investment Density"]
  E --> G["Outcomes: Better Applicability<br>vs. Continuous-Time Models"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a discrete-time mean‑variance reinforcement‑learning framework and an algorithm that finds the global optimal solution.

Source sentence
The first contribution of this paper is to present the global optimal solution to the discrete-time exploratory MV problem.

Assumes 5

  • The risky asset’s return distribution is not assumed to follow a geometric Brownian motion; weaker and more general assumptions are used. market
    Source sentence
    we want to highlight that without the assumption that the risky asset’s price process is a geometric Brownian motion, the EMV algorithm designed by Wang and Zhou (2020) may not work very well.
  • Excess returns of the risky asset follow a stationary skewed t distribution in the simulation study. model
    Source sentence
    We perform numerical simulation in a stationary market environment, where the excess return of the risky asset is simulated according to the skewed t distribution.
  • Monthly rebalancing over a 3‑month horizon (T = 3, Δt = 1). execution
    Source sentence
    We take T = 3 and Δt = 1, indicating that the MV problem is considered over a 3‑month period, with monthly rebalancing.
  • The exploration rate λ = 2 measures the trade‑off between exploitation and exploration. model
    Source sentence
    The exploration rate λ = 2 measures the trade‑off between exploitation and exploration in this MV problem.
Show 1 more
  • The risk‑free rate is constant at 2% and known. market
    Source sentence
    The annualized interest rate rf is taken to be 2%, which is known and fixed.

Data

  • S&P 500 index data market-prices public
    S&P 500 constituents
    Source sentence
    We backtest our algorithm using the S&P 500 index data in the empirical analysis.
  • Synthetic stationary market simulation synthetic unknown
    Source sentence
    We perform numerical simulation in a stationary market environment, where the excess return of the risky asset is simulated according to the skewed t distribution.

Costs

Not modeled A trading or execution result is reported without costs.

Finds 4

  • The proposed RL algorithm achieves better out‑of‑sample performance than the EMV algorithm in all simulations. performance
    Source sentence
    performances than the EMV algorithm in Wang and Zhou (2020) in all simulations.
  • The algorithm yields higher out‑of‑sample returns and converges relatively faster than the EMV algorithm on S&P 500 data. performance
    Source sentence
    We also find that our algorithm brings better out‑of‑sample returns and converges relatively faster than the EMV algorithm in Wang and Zhou (2020).
  • The optimal feedback control policy is Gaussian with a time‑decaying variance, implying decreasing exploration over time. theoretical
    Source sentence
    We also find that the optimal feedback control policy is Gaussian with a time‑decaying variance, which suggests that the level of exploration decreases as time passes.
  • A perfect separation between exploitation and exploration is achieved in the mean and variance of the optimal Gaussian policy. theoretical
    Source sentence
    Another interesting result is the perfect separation between exploitation and exploration in the mean and variance of the optimal Gaussian policy.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 43 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 2 papers in the archive (1 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (33)

  1. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation (2000)
  2. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  3. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  4. Portfolio selection (1952)
  5. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  6. Large scale continuous-time mean-variance portfolio allocation via reinforcement learning (2019)
  7. Financial data and the skewed generalized t distribution (1998)
  8. Mean-variance hedging in continuous time (1991)
and 25 more
  1. Optimal execution of portfolio transactions (2001)
  2. Recent advances in reinforcement learning in finance (2023)
  3. Algorithm trading using Q -learning and recurrent reinforcement learning (2016)
  4. Reinforcement learning in Portfolio Management and its interpretation (2018)
  5. Portfolio Management using Reinforcement Learning (2016)
  6. Algorithmic aspects of mean–variance optimization in Markov decision processes (2013)
  7. Model-free reinforcement learning for financial portfolios: a brief survey (2019) — link
  8. An intelligent financial portfolio trading strategy using deep Q -learning (2020)
  9. G-learner and girl: Goal based wealth management with reinforcement learning (2020) — link
  10. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  11. Practical deep reinforcement learning approach for stock trading (2018) — link
  12. Continuous control with deep reinforcement learning (2015) — link
  13. Adversarial deep reinforcement learning in portfolio management (2018) — link
  14. Reinforcement Learning for Continuous-Time Mean-Variance Portfolio Selection in a Regime-Switching Market (2023)
  15. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  16. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  17. q-Learning in Continuous Time (2023)
  18. Dynamic mean-variance portfolio selection with no-shorting constraints (2002)
  19. Optimal multi-period mean-variance policy under no-shorting constraint (2014)
  20. Regime-switching recurrent reinforcement learning for investment decision making (2012)
  21. Transition variable selection for regime switching recurrent reinforcement learning (2014)
  22. Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
  23. A reinforcement learning approach for solving the mean variance customer portfolio in partially observable models (2018)
  24. Survey on Multi-period Mean–Variance Portfolio Selection Model (2022)
  25. Learning equilibrium mean-variance strategy (2023)

0 of 33 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.