Paper: arXiv 2307.07694

Authors: Chung I Lu

Abstract

We evaluate benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using simulated data. The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. Using the Kelly criterion (log utility) as the objective, we can analytically derive the optimal policy without market impact as an upper bound to measure performance when including market impact. We find that the off-policy algorithms DDPG, TD3 and SAC are unable to learn the right \(Q\)-function due to the noisy rewards and therefore perform poorly. The on-policy algorithms PPO and A2C, with the use of generalised advantage estimation, are able to deal with the noise and derive a close to optimal policy. The clipping variant of PPO was found to be important in preventing the policy from deviating from the optimal once converged. In a more challenging environment where we have regime changes in the GBM parameters, we find that PPO, combined with a hidden Markov model to learn and predict the regime context, is able to learn different policies adapted to each regime. Overall, we find that the sample complexity of these algorithms is too high for applications using real data, requiring more than 2m steps to learn a good policy in the simplest setting, which is equivalent to almost 8,000 years of daily prices.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics, including stochastic calculus (Itô’s lemma), SDEs, and regime switching models, while also using a well-defined simulated environment and analytical benchmarks to rigorously evaluate algorithm performance, though it lacks real-world data backtesting.

Research Flowchart

  flowchart TD
  A["Research Goal: Evaluate DRL Algorithms<br>for Portfolio Optimisation"] --> B{"Methodology"}
  
  B --> C["Simulation: GBM +<br>Bertsimas-Lo Market Impact"]
  B --> D["Objective: Kelly Criterion<br>Log Utility"]
  B --> E["Benchmark: Analytical<br>Optimal Policy"]
  
  C & D --> F["Algorithm Evaluation"]
  
  F --> G["Off-Policy Failure<br>DDPG/TD3/SAC<br>Noisy Rewards"]
  F --> H["On-Policy Success<br>PPO/A2C + GAE<br>Close to Optimal"]
  
  H --> I["Key Findings"]
  
  I --> J["PPO with HMM adapts<br>to regime changes"]
  I --> K["High sample complexity<br>>2M steps equivalent<br>to ~8000 years"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper evaluates benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using a simulator based on correlated geometric Brownian motion and the Bertsimas‑Lo market impact model.

Source sentence
We evaluate benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using simulated data.

Assumes 5

  • Asset prices follow correlated geometric Brownian motion. model
    Source sentence
    The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. 3.1.1 Geometric Brownian Motion The simulator generates time series of prices for the stocks using correlated geometric Brownian motion (GBM) [13, Chapter 5.4.2].
  • Market impact is modeled using the Bertsimas‑Lo model. market
    Source sentence
    3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.
  • When market impact is set to zero, the analytically derived optimal policy serves as an upper bound. model
    Source sentence
    Using the Kelly criterion (log utility) as the objective, we can analytically derive the optimal policy without market impact as an upper bound to measure performance when including market impact. When including market impact, the derived optimal policy without market impact serves as an upper bound to measure performance.
  • The environment is non‑changing, leading to a fixed‑weight optimal policy. other
    Source sentence
    Due to the assumption of a non-changing environment, the optimal policy is a fixed weight portfolio, i.e., the weights do not change over time.
Show 1 more
  • Initial portfolio wealth is low (e.g., 1,000) resulting in low market impact in certain experiments. execution
    Source sentence
    6 Experiments 6.1 Low Initial Wealth We consider experiments with low initial wealth of 1,000 which results in low market impact. The parameters for the GBM, market impact and algorithms are provided in Appendix C. In this setting, the market impact is low and we expect the optimal policy to be close that described in Section 4.1.

Data

  • Simulated price data synthetic unknown
    three correlated stocks and a cash account
    Source sentence
    The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. 3.1 Simulator The simulator has three components: 1.
  • ETF price data (VUG, VTV, GLD) · Yahoo Finance market-prices public
    2018‑2023 · VUG, VTV, GLD
    Source sentence
    The time step ∆t = 1 represents one year while each trading day is represented by a time step of ∆t = 1 252. The ETFs are chosen as representations of portfolio allocation components with VUG being a growth stocks ETF, VTV being a value stocks ETF and GLD being a gold ETF.

Recognised sources: Yahoo Finance

Costs

Modeled Costs are part of the reported headline results.

Transaction cost and market impact are incorporated via the Bertsimas‑Lo market impact model.

Source sentence
3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.

Finds 5

  • Off‑policy algorithms (DDPG, TD3, SAC) are unable to learn the correct Q‑function due to noisy rewards and perform poorly. negative
    Source sentence
    We find that the off-policy algorithms DDPG, TD3 and SAC are unable to learn the right $Q$-function due to the noisy rewards and therefore perform poorly.
  • On‑policy algorithms (PPO, A2C) with generalized advantage estimation can handle reward noise and learn a near‑optimal policy. performance
    Source sentence
    The on-policy algorithms PPO and A2C, with the use of generalised advantage estimation, are able to deal with the noise and derive a close to optimal policy.
  • The clipping variant of PPO prevents the policy from drifting away from the optimal once converged. performance
    Source sentence
    The clipping variant of PPO was found to be important in preventing the policy from deviating from the optimal once converged. The clipping function of PPO reduces the likelihood of deviating from the optimal policy once it is found. 3.
  • Sample complexity is very high; the best algorithm (PPO) requires more than 2 million steps (~8,000 years of daily data) to learn a good policy in the simplest setting. performance
    Source sentence
    Overall, we find that the sample complexity of these algorithms is too high for applications using real data, requiring more than 2m steps to learn a good policy in the simplest setting, which is equivalent to almost 8,000 years of daily prices. It takes at least 2m steps with the best algorithm (PPO) to converge to a good policy 17 Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation
  • In a regime‑changing environment, PPO combined with a hidden Markov model can learn distinct policies adapted to each regime. performance
    Source sentence
    In a more challenging environment where we have regime changes in the GBM parameters, we find that PPO, combined with a hidden Markov model to learn and predict the regime context, is able to learn different policies adapted to each regime. This allowed the PPO algorithm to adapt different policies to each regime.

Checked by others

Supported by

  • On-line reinforcement learning for optimization of real-life energy trading strategy (2023)
    On the question: Does the SAC/off‑policy algorithm achieve profitable and stable performance in real‑world energy trading and portfolio optimisation?
    Compare the findings

    This paper: Off‑policy algorithms (DDPG, TD3, SAC) are unable to learn the correct Q‑function due to noisy rewards and perform poorly.

    That paper: The SAC algorithm performs worst, with average profit around zero and highly unstable behavior.

    Both findings report poor outcomes: SAC yields average profit ≈0 with high instability, and off‑policy methods (DDPG, TD3, SAC) fail to learn correct Q‑functions due to noisy rewards.

  • Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity (2024)
    On the question: Which deep reinforcement learning algorithm achieves the highest cumulative reward for portfolio optimisation?
    Compare the findings

    This paper: On‑policy algorithms (PPO, A2C) with generalized advantage estimation can handle reward noise and learn a near‑optimal policy.

    That paper: A2C emerges as the top performer in terms of cumulative rewards among the DRL algorithms studied.

    Both findings highlight A2C as the top performer; A notes on‑policy methods (PPO, A2C) handle reward noise, B confirms A2C leads in cumulative rewards.

Cited by 2 papers in the archive (1 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.