Paper: arXiv 2502.02619

Authors: Daniil Karzanov, Rubén Garzón, Mikhail Terekhov, Caglar Gulcehre, Thomas Raffinot, Marcin Detyniecki

Abstract

This paper introduces a novel agent-based approach for enhancing existing portfolio strategies using Proximal Policy Optimization (PPO). Rather than focusing solely on traditional portfolio construction, our approach aims to improve an already high-performing strategy through dynamic rebalancing driven by PPO and Oracle agents. Our target is to enhance the traditional 60/40 benchmark (60% stocks, 40% bonds) by employing the Regret-based Sharpe reward function. To address the impact of transaction fee frictions and prevent signal loss, we develop a transaction cost scheduler. We introduce a future-looking reward function and employ synthetic data training through a circular block bootstrap method to facilitate the learning of generalizable allocation strategies. We focus on two key evaluation measures: return and maximum drawdown. Given the high stochasticity of financial markets, we train 20 independent agents each period and evaluate their average performance against the benchmark. Our method not only enhances the performance of the existing portfolio strategy through strategic rebalancing but also demonstrates strong results compared to other baselines.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 6.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced deep reinforcement learning techniques like PPO with custom regret-based reward functions and synthetic data generation, resulting in high mathematical complexity. It demonstrates strong empirical rigor through specific backtest metrics, transaction cost modeling, and robust evaluation across training, validation, and test sets with ablation studies.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Enhance 60/40 Portfolio using<br>Deep Reinforcement Learning"] --> B["Data Input<br>Synthetic Data via<br>Circular Block Bootstrap"]
  
  B --> C["Methodology<br>Proximal Policy Optimization<br>with Regret-based Sharpe Reward"]
  
  C --> D["Key Innovation<br>Future-Looking Reward &<br>Transaction Cost Scheduler"]
  
  D --> E["Computation<br>Train 20 Independent Agents<br>Average Stochastic Performance"]
  
  E --> F["Outcomes<br>Enhanced Returns & Reduced<br>Max Drawdown vs. Baselines"]
  
  F --> G["Conclusion<br>Effective Dynamic Rebalancing<br>with Strong Generalization"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a novel agent‑based PPO approach to dynamically rebalance and enhance an existing high‑performing 60/40 portfolio using a regret‑based Sharpe reward and a transaction‑cost scheduler.

Source sentence
Our target is to enhance the traditional 60/40 benchmark (60% stocks, 40% bonds) by employing the Regret‑based Sharpe reward function. To address the impact of transaction fee frictions and prevent signal loss, we develop a transaction cost scheduler.

Assumes 5

  • The environment has no price impact from the agent’s allocations. execution
    Source sentence
    our environment lacks price impact and remains unaffected by allocations.
  • Future return information is unavailable at inference time to avoid data leakage. data
    Source sentence
    Since we do not have access to future return information during testing, we set the reward to zero in the environment when deploying the trained model.
  • Transaction costs are gradually introduced during training via a scheduler. execution
    Source sentence
    we introduce a transaction cost scheduler that incrementally raises the training transaction cost at each step according to 𝑇𝐶train(𝑥) = 𝑇𝐶max 𝑆𝑎·𝑥𝑎 if 0 ≤𝑥≤𝑆.
  • Synthetic training data generated by circular block bootstrap approximates the real data distribution. data
    Source sentence
    we opted for a circular block bootstrap [37] of the underlying training data, applied every 10 episodes.
Show 1 more
  • The agent can observe a vector of current returns, averages, volatilities, previous weights and transaction costs. model
    Source sentence
    At each timestep, the agent observes a vector 𝑜𝑡= [𝜇𝑡, 𝛼𝑡, ¯𝜇𝑡−40𝑡, ¯𝜎𝑡−60𝑡, ¯𝑞𝑡−60𝑡, 𝑤𝑡−1, 𝑇𝐶𝑡𝑟𝑎𝑖𝑛(𝑡)] ∈R19.

Data

  • Daily price data for assets and index tickers market-prices unknown
    Source sentence
    We transform the price data for both asset and index tickers into daily strategy returns, 𝜇𝑡∈R3 and 𝛼𝑡∈R3 respectively...
  • Synthetic data via circular block bootstrap synthetic unknown
    Source sentence
    we opted for a circular block bootstrap [37] of the underlying training data, applied every 10 episodes.
  • 60/40 benchmark (60% stocks, 40% bonds) benchmark unknown
    Source sentence
    Our target is to enhance the traditional 60/40 benchmark (60% stocks, 40% bonds)...

Costs

Modeled Costs are part of the reported headline results.

Costs are ramped up during training and fixed at 0.0025 (0.25%) for evaluation.

Source sentence
After the ramp limit, the maximal costs, 𝑇𝐶max = 𝑇𝐶eval = 0.0025 (a fair value for traditional brokers and large institutional traders), are applied.

Finds 3

  • The PPO agent outperforms the 60/40 portfolio in return across all periods except the validation set in the pre‑pandemic phase. performance
    Source sentence
    Our PPO agent outperforms the 60/40 portfolio in terms of return across all periods, except for the validation set in the pre‑pandemic phase.
  • Regret‑PPO achieves a significantly higher Calmar ratio in most cases, though not in Phase 2. performance
    Source sentence
    The Calmar ratio is significantly higher for regret PPO in most cases, with the exception of Phase 2.
  • Embedded drawdown and differential Sharpe reward functions are more conservative, performing well on risk‑adjusted metrics such as Sharpe, Sortino, and MDD. performance
    Source sentence
    Both the embedded drawdown and differential Sharpe reward functions are more conservative, consistently performing well in risk‑accounting measures such as Sharpe, Sortino, and MDD.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 22 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (50)

  1. Combining Reinforcement Learning and Inverse Reinforcement Learning for Asset Allocation Recommendations — link
  2. Headline-Driven Classification and Local Interpretation for Market Outperformance and Low-Risk Stock Prediction (2023)
  3. Machine learning for asset managers (2020)
  4. It’s a Peoples Game, Isn’t It?! A Comparison Between the Investment Returns of Business Angels and Machine Learning Algorithms (2020) — link
  5. High-Dimensional Stock Portfolio Trading with Deep Reinforcement Learning — link
  6. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading — link
  7. Practical Deep Reinforcement Learning Approach for Stock Trading — link
  8. Deep Reinforcement Learning for Asset Allocation in US Equities — link
and 42 more
  1. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem — link
  2. Embedded draw-down constraint reward function for deep reinforcement learning — link
  3. Headline-Driven Classification and Local Interpretation for Market Outperformance and Low-Risk Stock Prediction — link
  4. Combining deep reinforcement learning with technical analysis and trend monitoring on cryptocurrency markets (2023)
  5. Bridging the gap between Markowitz planning and deep reinforcement learning (2020) — link
  6. Embedded draw-down constraint reward function for deep reinforcement learning (2022)
  7. Deep reinforcement learning for asset allocation in us equities (2020) — link
  8. Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation — → 2307.07694
  9. Detecting and adapting to crisis pattern with context based Deep Reinforcement Learning (2021)
  10. Generative adversarial imitation learning (2016)
  11. Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2018) — link
  12. Remarks on some nonparametric estimates of a density function (2011)
  13. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  14. High-dimensional stock portfolio trading with deep reinforcement learning (2022)
  15. Deep reinforcement trading with predictable returns (2023)
  16. Deep reinforcement learning based multi-objective systems for financial trading (2020)
  17. Multi-objective reward generalization: improving performance of Deep Reinforcement Learning for applications in single-asset trading (2024)
  18. Recent advances in reinforcement learning in finance (2023)
  19. Review of multi-criteria decision-making methods in finance using explainable artificial intelligence (2022)
  20. An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown (2017)
  21. Measuring regret theory in the health and financial domain (2023)
  22. A deep Q-learning portfolio management framework for the cryptocurrency market (2020)
  23. A deep reinforcement learning approach for automated cryptocurrency trading (2019)
  24. Copula models in machine learning (2015)
  25. ARCH: Autoregressive Conditional Heteroskedasticity models in Python (2023) — link
  26. Human-level control through deep reinforcement learning (2015)
  27. Proximal policy optimization algorithms (2017) — link
  28. Mastering the game of Go with deep neural networks and tree search (2016)
  29. Reinforcement learning: An introduction (2018)
  30. Portfolio Selection (1952)
  31. Investor-imitator: A framework for trading knowledge extraction (2018)
  32. A modified CTGAN-plus-features-based method for optimal asset allocation — → 2302.02269
  33. A synthetic data-plus-features driven approach for portfolio optimization (2023)
  34. Deep Reinforcement Learning for Optimal Portfolio Allocation: A Comparative Study with Mean-Variance Optimization (2023)
  35. Credit spreads and business cycle fluctuations (2012)
  36. The investor fear gauge (2000)
  37. The price of correlation risk: Evidence from equity options (2009)
  38. Curriculum learning (2009)
  39. Curriculum learning in deep neural networks for financial forecasting (2020)
  40. Drawdown measure in portfolio optimization (2005)
  41. Smart “predict, then optimize” (2022)
  42. Smart predict-and-optimize for hard combinatorial optimization problems (2020)

2 of 50 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.