{"paper_id":"2307.07694","title":"Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation","authors":["Chung I Lu"],"paper_date":"2023-07-15","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2307.07694v3","page_url":"https://thequant.space/flowcharts/evaluation-of-deep-reinforcement-learning-algorithms-for-por/","scores":{"empirical_rigor":6,"math_complexity":7.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper evaluates benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using a simulator based on correlated geometric Brownian motion and the Bertsimas‑Lo market impact model.","quote":"We evaluate benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using simulated data.","summary":"We evaluate benchmark deep reinforcement learning algorithms on the task of portfolio optimisation using simulated data. The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. Using the Kelly criterion (log utility) as the…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Asset prices follow correlated geometric Brownian motion.","type":"model","stated":"explicit","quote":"The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. 3.1.1 Geometric Brownian Motion The simulator generates time series of prices for the stocks using correlated geometric Brownian motion (GBM) [13, Chapter 5.4.2].","support":"full"},{"assumption":"Market impact is modeled using the Bertsimas‑Lo model.","type":"market","stated":"explicit","quote":"3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.","support":"full"},{"assumption":"When market impact is set to zero, the analytically derived optimal policy serves as an upper bound.","type":"model","stated":"explicit","quote":"Using the Kelly criterion (log utility) as the objective, we can analytically derive the optimal policy without market impact as an upper bound to measure performance when including market impact. When including market impact, the derived optimal policy without market impact serves as an upper bound to measure performance.","support":"full"},{"assumption":"The environment is non‑changing, leading to a fixed‑weight optimal policy.","type":"other","stated":"explicit","quote":"Due to the assumption of a non-changing environment, the optimal policy is a fixed weight portfolio, i.e., the weights do not change over time.","support":"full"},{"assumption":"Initial portfolio wealth is low (e.g., 1,000) resulting in low market impact in certain experiments.","type":"execution","stated":"explicit","quote":"6 Experiments 6.1 Low Initial Wealth We consider experiments with low initial wealth of 1,000 which results in low market impact. The parameters for the GBM, market impact and algorithms are provided in Appendix C. In this setting, the market impact is low and we expect the optimal policy to be close that described in Section 4.1.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Simulated price data","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":"three correlated stocks and a cash account","quote":"The simulator to generate the data is based on correlated geometric Brownian motion with the Bertsimas-Lo market impact model. 3.1 Simulator The simulator has three components: 1.","support":"full"},{"name":"ETF price data (VUG, VTV, GLD)","provider":"Yahoo Finance","kind":"market-prices","access":"public","period":"2018‑2023","universe":"VUG, VTV, GLD","quote":"The time step ∆t = 1 represents one year while each trading day is represented by a time step of ∆t = 1 252. The ETFs are chosen as representations of portfolio allocation components with VUG being a growth stocks ETF, VTV being a value stocks ETF and GLD being a gold ETF.","support":"partial","known_id":"yahoo-finance"}],"known":[{"id":"yahoo-finance","name":"Yahoo Finance","provider":"Yahoo","kind":"market-prices","access":"public","quote":"The data was obtained from Yahoo Finance from 2018 to 2023.","mentions":1,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":"Transaction cost and market impact are incorporated via the Bertsimas‑Lo market impact model.","quote":"3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.","source":"front_matter","rule":"R3","candidates":{"front_matter":"yes","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Transaction cost and market impact are incorporated via the Bertsimas‑Lo market impact model.","card_quote":"3.1.2 Bertsimas-Lo Market Impact Model A GBM simulator alone lacks two important aspects of the real world: transaction cost and market impact. Therefore, we will use the Bertsimas-Lo (BL) model [14], which uses GBM as the base price process, to model the impact of a trade on the price of the asset.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"Off‑policy algorithms (DDPG, TD3, SAC) are unable to learn the correct Q‑function due to noisy rewards and perform poorly.","kind":"negative","quote":"We find that the off-policy algorithms DDPG, TD3 and SAC are unable to learn the right $Q$-function due to the noisy rewards and therefore perform poorly.","support":"full"},{"finding":"On‑policy algorithms (PPO, A2C) with generalized advantage estimation can handle reward noise and learn a near‑optimal policy.","kind":"performance","quote":"The on-policy algorithms PPO and A2C, with the use of generalised advantage estimation, are able to deal with the noise and derive a close to optimal policy.","support":"full"},{"finding":"The clipping variant of PPO prevents the policy from drifting away from the optimal once converged.","kind":"performance","quote":"The clipping variant of PPO was found to be important in preventing the policy from deviating from the optimal once converged. The clipping function of PPO reduces the likelihood of deviating from the optimal policy once it is found. 3.","support":"full"},{"finding":"Sample complexity is very high; the best algorithm (PPO) requires more than 2 million steps (~8,000 years of daily data) to learn a good policy in the simplest setting.","kind":"performance","quote":"Overall, we find that the sample complexity of these algorithms is too high for applications using real data, requiring more than 2m steps to learn a good policy in the simplest setting, which is equivalent to almost 8,000 years of daily prices. It takes at least 2m steps with the best algorithm (PPO) to converge to a good policy 17 Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation","support":"full"},{"finding":"In a regime‑changing environment, PPO combined with a hidden Markov model can learn distinct policies adapted to each regime.","kind":"performance","quote":"In a more challenging environment where we have regime changes in the GBM parameters, we find that PPO, combined with a hidden Markov model to learn and predict the regime context, is able to learn different policies adapted to each regime. This allowed the PPO algorithm to adapt different policies to each regime.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[{"paper":{"paper_id":"2303.16266","title":"On-line reinforcement learning for optimization of real-life energy trading strategy","paper_date":"2023-03-28","authors":["Łukasz Lepak","Paweł Wawrzyński"],"empirical_rigor":6.5,"page_url":"https://thequant.space/flowcharts/on-line-reinforcement-learning-for-optimization-of-real-life-energy-trading/"},"question":"Does the SAC/off‑policy algorithm achieve profitable and stable performance in real‑world energy trading and portfolio optimisation?","this_paper_finding":"Off‑policy algorithms (DDPG, TD3, SAC) are unable to learn the correct Q‑function due to noisy rewards and perform poorly.","other_paper_finding":"The SAC algorithm performs worst, with average profit around zero and highly unstable behavior.","other_paper_quote":"the worst of all tested algorithms is SAC, which profit averages around 0 with very unstable behavior – some runs generate some profit, while others end up with significant losses.","note":"Both findings report poor outcomes: SAC yields average profit ≈0 with high instability, and off‑policy methods (DDPG, TD3, SAC) fail to learn correct Q‑functions due to noisy rewards.","confidence":0.721},{"paper":{"paper_id":"2407.09557","title":"Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity","paper_date":"2024-06-29","authors":["Alireza Mohammadshafie","Akram Mirzaeinia","Haseebullah Jumakhan","Amir Mirzaeinia"],"empirical_rigor":7,"page_url":"https://thequant.space/flowcharts/deep-reinforcement-learning-strategies-in-finance-insights/"},"question":"Which deep reinforcement learning algorithm achieves the highest cumulative reward for portfolio optimisation?","this_paper_finding":"On‑policy algorithms (PPO, A2C) with generalized advantage estimation can handle reward noise and learn a near‑optimal policy.","other_paper_finding":"A2C emerges as the top performer in terms of cumulative rewards among the DRL algorithms studied.","other_paper_quote":"Our findings reveal that each DRL algorithm exhibits unique trading patterns and strategies, with A2C emerging as the top performer in terms of cumulative rewards.","note":"Both findings highlight A2C as the top performer; A notes on‑policy methods (PPO, A2C) handle reward noise, B confirms A2C leads in cumulative rewards.","confidence":0.7}],"contradicts":[],"qualifies":[],"compared_with":17,"citations":{"cited_by_in_corpus":2,"by_independent_authors":1,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":1,"unclassified":1},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T20:02:06Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}