Paper: arXiv 2307.07694
Authors: Chung I Lu
Abstract
We evaluate benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using simulated data. The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. Using the Kelly criterion (log utility) as the objective, we can analytically derive the optimal policy without market impact as an upper bound to measure performance when including market impact. We find that the off-policy algorithms DDPG, TD3 and SAC are unable to learn the right \(Q\)-function due to the noisy rewards and therefore perform poorly. The on-policy algorithms PPO and A2C, with the use of generalised advantage estimation, are able to deal with the noise and derive a close to optimal policy. The clipping variant of PPO was found to be important in preventing the policy from deviating from the optimal once converged. In a more challenging environment where we have regime changes in the GBM parameters, we find that PPO, combined with a hidden Markov model to learn and predict the regime context, is able to learn different policies adapted to each regime. Overall, we find that the sample complexity of these algorithms is too high for applications using real data, requiring more than 2m steps to learn a good policy in the simplest setting, which is equivalent to almost 8,000 years of daily prices.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs advanced mathematics, including stochastic calculus (Itô’s lemma), SDEs, and regime switching models, while also using a well-defined simulated environment and analytical benchmarks to rigorously evaluate algorithm performance, though it lacks real-world data backtesting.
Research Flowchart
flowchart TD
A["Research Goal: Evaluate DRL Algorithms<br>for Portfolio Optimisation"] --> B{"Methodology"}
B --> C["Simulation: GBM +<br>Bertsimas-Lo Market Impact"]
B --> D["Objective: Kelly Criterion<br>Log Utility"]
B --> E["Benchmark: Analytical<br>Optimal Policy"]
C & D --> F["Algorithm Evaluation"]
F --> G["Off-Policy Failure<br>DDPG/TD3/SAC<br>Noisy Rewards"]
F --> H["On-Policy Success<br>PPO/A2C + GAE<br>Close to Optimal"]
H --> I["Key Findings"]
I --> J["PPO with HMM adapts<br>to regime changes"]
I --> K["High sample complexity<br>>2M steps equivalent<br>to ~8000 years"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper evaluates benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using a simulator based on correlated geometric Brownian motion and the Bertsimas‑Lo market impact model.
Source sentence
We evaluate benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using simulated data.
Assumes 5
- Asset prices follow correlated geometric Brownian motion. model
Source sentence
The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. 3.1.1 Geometric Brownian Motion The simulator generates time series of prices for the stocks using correlated geometric Brownian motion (GBM) [13, Chapter 5.4.2].
- Market impact is modeled using the Bertsimas‑Lo model. market
Source sentence
3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.
- When market impact is set to zero, the analytically derived optimal policy serves as an upper bound. model
Source sentence
Using the Kelly criterion (log utility) as the objective, we can analytically derive the optimal policy without market impact as an upper bound to measure performance when including market impact. When including market impact, the derived optimal policy without market impact serves as an upper bound to measure performance.
- The environment is non‑changing, leading to a fixed‑weight optimal policy. other
Source sentence
Due to the assumption of a non-changing environment, the optimal policy is a fixed weight portfolio, i.e., the weights do not change over time.
Show 1 more
- Initial portfolio wealth is low (e.g., 1,000) resulting in low market impact in certain experiments. execution
Source sentence
6 Experiments 6.1 Low Initial Wealth We consider experiments with low initial wealth of 1,000 which results in low market impact. The parameters for the GBM, market impact and algorithms are provided in Appendix C. In this setting, the market impact is low and we expect the optimal policy to be close that described in Section 4.1.
Data
- Simulated price data synthetic unknown
Source sentence
The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. 3.1 Simulator The simulator has three components: 1.
- ETF price data (VUG, VTV, GLD) market-prices public
Source sentence
The time step ∆t = 1 represents one year while each trading day is represented by a time step of ∆t = 1 252. The ETFs are chosen as representations of portfolio allocation components with VUG being a growth stocks ETF, VTV being a value stocks ETF and GLD being a gold ETF.
Costs
Modeled
Transaction cost and market impact are incorporated via the Bertsimas‑Lo market impact model.
Source sentence
3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.
Finds 5
- Off‑policy algorithms (DDPG, TD3, SAC) are unable to learn the correct Q‑function due to noisy rewards and perform poorly. negative
Source sentence
We find that the off-policy algorithms DDPG, TD3 and SAC are unable to learn the right $Q$-function due to the noisy rewards and therefore perform poorly.
- On‑policy algorithms (PPO, A2C) with generalized advantage estimation can handle reward noise and learn a near‑optimal policy. performance
Source sentence
The on-policy algorithms PPO and A2C, with the use of generalised advantage estimation, are able to deal with the noise and derive a close to optimal policy.
- The clipping variant of PPO prevents the policy from drifting away from the optimal once converged. performance
Source sentence
The clipping variant of PPO was found to be important in preventing the policy from deviating from the optimal once converged. The clipping function of PPO reduces the likelihood of deviating from the optimal policy once it is found. 3.
- Sample complexity is very high; the best algorithm (PPO) requires more than 2 million steps (~8,000 years of daily data) to learn a good policy in the simplest setting. performance
Source sentence
Overall, we find that the sample complexity of these algorithms is too high for applications using real data, requiring more than 2m steps to learn a good policy in the simplest setting, which is equivalent to almost 8,000 years of daily prices. It takes at least 2m steps with the best algorithm (PPO) to converge to a good policy 17 Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation
- In a regime‑changing environment, PPO combined with a hidden Markov model can learn distinct policies adapted to each regime. performance
Source sentence
In a more challenging environment where we have regime changes in the GBM parameters, we find that PPO, combined with a hidden Markov model to learn and predict the regime context, is able to learn different policies adapted to each regime. This allowed the PPO algorithm to adapt different policies to each regime.
Checked by others
Supported by
- On-line reinforcement learning for optimization of real-life energy trading strategy
Compare the findings
This paper: Off‑policy algorithms (DDPG, TD3, SAC) are unable to learn the correct Q‑function due to noisy rewards and perform poorly.
That paper: The SAC algorithm performs worst, with average profit around zero and highly unstable behavior.
- Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity
Compare the findings
This paper: On‑policy algorithms (PPO, A2C) with generalized advantage estimation can handle reward noise and learn a near‑optimal policy.
That paper: A2C emerges as the top performer in terms of cumulative rewards among the DRL algorithms studied.
2 papers in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.