Paper: arXiv 2610.01325
Authors: Duong Hien Chi Kien, Thanh Trung Huynh
Abstract
Reinforcement learning for trading often struggles to balance upside participation with drawdown control. Profit-only policies can collapse toward passive long exposure on upward-drifting assets, while aggressively risk-penalized rewards can become too defensive during volatile periods. This paper proposes PPO-HRAP, a hybrid regime-aware policy that combines Proximal Policy Optimization with an interpretable regime prior. The agent observes both market features and portfolio-state variables, receives a reward combining portfolio log return, VIX-conditioned drawdown-increase penalty, target-exposure deviation, and turnover cost, and executes a blended action between the PPO actor output and a regime-derived target exposure. On the held-out 2020-2022 SPY test window, PPO-HRAP achieves 27.62% total return, 8.48% annualized return, 0.6447 Sharpe ratio, 0.8588 Sortino ratio, and 0.4592 Calmar ratio, while reducing maximum drawdown from 34.10% for Buy and Hold to 18.47%. Across five SPY seeds, PPO-HRAP remains stable with mean total return \(0.2725 \pm 0.0109\) and mean Sharpe ratio \(0.6219 \pm 0.0565\). Single-run cross-asset tests on QQQ and DIA further show that the proposed method ranks first on total return and Sharpe ratio for all three reported assets. These results suggest that blending learned actions with a volatility-aware regime prior is a practical way to improve risk-adjusted trading behavior, although the current policy still incurs high turnover and cross-asset robustness beyond SPY remains limited to single-run evidence.
Complexity vs Empirical Score
- Math Complexity: 6.0/10
- Empirical Rigor: 7.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper presents a novel hybrid RL approach with a clear methodology and strong empirical results on a relevant dataset. While the mathematical derivations are present, the focus is more on the practical application and empirical validation, which is well-executed.
Research Flowchart
flowchart TD
A["Research Goal: Improve Risk-Controlled Trading via RL"] --> B{"Key Methodology: PPO-HRAP"};
B --> C["Data/Inputs: Market Features, Portfolio State, VIX, Price Data (SPY, QQQ, DIA)"];
C --> D{"Computational Process: PPO Algorithm, Hybrid Policy Blending, Reward Function Optimization"};
D --> E["Outcomes: High Risk-Adjusted Returns (Sharpe 0.64), Reduced Max Drawdown (18.47% vs 34.10% BH), Cross-Asset Performance"];
E --> F["Limitations: High Turnover, Limited Cross-Asset Robustness (Single-Run Evidence)"];
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes PPO-HRAP, a hybrid regime-aware policy that combines Proximal Policy Optimization with an interpretable regime prior.
Source sentence
This paper proposes PPO-HRAP, a hybrid regime-aware policy that combines Proximal Policy Optimization with an interpretable regime prior.
Assumes 5
- All methods are evaluated with a 0.1% proportional transaction cost. execution
Source sentence
All methods use a 0.1% proportional transaction cost.
- Data are split chronologically: 2010–2017 for training, 2018–2019 for validation, and 2020–2022 for testing. data
Source sentence
We use chronological splits: 2010–2017 for training, 2018–2019 for validation, and 2020–2022 for testing.
- The VIX z‑score is used to scale the dynamic drawdown penalty. market
Source sentence
Thus, when VIX z-score rises, the drawdown penalty increases; when stress is low, the penalty relaxes toward its base level.
- The main SPY result uses random seed 42. model
Source sentence
The main SPY result uses seed 42
Show 1 more
- Action‑prior weight ρ is set to 0.6 based on validation. model
Source sentence
The validation search over ρ selects ρ= 0.6, with validation Calmar 0.2299 and validation total return 0.0800.
Data
2020 – 2022 · SPY (S&P 500 ETF) · equities · daily
- SPY market-prices public
Source sentence
On the held-out 2020–2022 SPY test window, PPO-HRAP achieves 27.62% total return...
- QQQ market-prices public
Source sentence
Single-run cross-asset tests on QQQ and DIA further show that the proposed method ranks first on total return and Sharpe ratio for all three reported assets.
- DIA market-prices public
Source sentence
Single-run cross-asset tests on QQQ and DIA further show that the proposed method ranks first on total return and Sharpe ratio for all three reported assets.
Costs
Modeled
0.1% proportional transaction cost applied to all methods
Source sentence
All methods use a 0.1% proportional transaction cost.
Finds 5
Sharpe 0.64 (benchmark 0.34) · annual return 8.5% · total return 27.6% · max drawdown 18.5%
- PPO-HRAP achieves 27.62% total return, 8.48% annualized return, 0.6447 Sharpe, 0.8588 Sortino, 0.4592 Calmar, and reduces maximum drawdown from 34.10% (Buy and Hold) to 18.47% on SPY 2020–2022. performance
Source sentence
PPO-HRAP achieves 27.62% total return, 8.48% annualized return, 0.6447 Sharpe ratio, 0.8588 Sortino ratio, and 0.4592 Calmar ratio, while reducing maximum drawdown from 34.10% for Buy and Hold to 18.47%.
- Across five SPY seeds, PPO-HRAP shows mean total return $0.2725 \pm 0.0109$ and mean Sharpe $0.6219 \pm 0.0565$, indicating stability. performance
Source sentence
Across five SPY seeds, PPO-HRAP remains stable with mean total return $0.2725 \pm 0.0109$ and mean Sharpe ratio $0.6219 \pm 0.0565$.
- PPO-HRAP ranks first on total return and Sharpe ratio for SPY, QQQ, and DIA in single-run cross‑asset tests. performance
Source sentence
PPO-HRAP ranks first on total return and Sharpe ratio for SPY, QQQ, and DIA.
- Action‑prior weight ρ=0.6 yields validation Calmar 0.2299 and validation total return 0.0800; performance is non‑monotonic in ρ. other
Source sentence
The validation search over ρ selects ρ= 0.6, with validation Calmar 0.2299 and validation total return 0.0800.
- The smallest drawdown (9.51%) is achieved by PPO Static MDD but with much lower total return (5.91%). performance
Source sentence
The smallest drawdown is obtained by PPO Static MDD (9.51%), but at the cost of a much lower total return (5.91%).
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 43 related papers). Coverage is partial, so this is neither replication nor refutation.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (43)
- Proximal Policy Optimization Algorithms (2017) — link
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018)
- A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
- MTS: A Deep Reinforcement Learning Portfolio Management Framework with Time-Awareness and Short-Selling (2025) — link
- The Sharpe Ratio (1994)
- Performance Measurement in a Downside Risk Framework (1994)
- Understanding the VIX (2009)
- New Concepts in Technical Trading Systems (1978)
and 35 more
- Bollinger on Bollinger Bands (2002)
- On the Properties of Equally-Weighted Risk Contributions Portfolios (2008)
- Theory of Constant Proportion Portfolio Insurance (1992)
- Portfolio Selection (1952)
- Optimum Consumption and Portfolio Rules in a Continuous-Time Model (1971)
- A New Interpretation of Information Rate (1956)
- Universal Portfolios (1991)
- On-Line Portfolio Selection Using Multiplicative Updates (1998)
- Performance Functions and Reinforcement Learning for Trading Systems and Portfolios (1998)
- Learning to Trade via Direct Reinforcement (2001)
- Reinforcement Learning for Optimized Trade Execution (2006)
- Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2017)
- An Adaptive Portfolio Trading System: A Risk-Return Portfolio Optimization Using Recurrent Reinforcement Learning with Expected Maximum Drawdown (2017)
- Adversarial Deep Reinforcement Learning in Portfolio Management (2018) — link
- Reinforcement-Learning Based Portfolio Management with Augmented Asset Movement Prediction States (2020)
- FinRL : A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance (2020)
- Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (2020)
- FinRL : Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021)
- FinRL-Meta : Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022)
- A Deep Q-Learning Portfolio Management Framework for the Cryptocurrency Market (2020)
- Deep Reinforcement Learning for Trading–A Critical Survey (2021)
- Deep Reinforcement Learning for Portfolio Selection (2024)
- Risk-Constrained Kelly Gambling (2016)
- Empirical Properties of Asset Returns: Stylized Facts and Statistical Issues (2001)
- Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation (1982)
- Generalized Autoregressive Conditional Heteroskedasticity (1986)
- Optimization of Conditional Value-at-Risk (2000)
- Drawdown Measure in Portfolio Optimization (2005)
- Human-Level Control Through Deep Reinforcement Learning (2015)
- Policy Gradient Methods for Reinforcement Learning with Function Approximation (1999)
- Reinforcement Learning: An Introduction (2018)
- Deterministic Policy Gradient Algorithms (2014)
- Continuous Control with Deep Reinforcement Learning (2015) — link
- Deep Learning with Long Short-Term Memory Networks for Financial Market Predictions (2018)
- Simple Technical Trading Rules and the Stochastic Properties of Stock Returns (1992)
0 of 43 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.