Paper: arXiv 2410.14927

Authors: Zijie Zhao, Roy E. Welsch

Abstract

Leveraging Deep Reinforcement Learning (DRL) in automated stock trading has shown promising results, yet its application faces significant challenges, including the curse of dimensionality, inertia in trading actions, and insufficient portfolio diversification. Addressing these challenges, we introduce the Hierarchical Reinforced Trader (HRT), a novel trading strategy employing a bi-level Hierarchical Reinforcement Learning framework. The HRT integrates a Proximal Policy Optimization (PPO)-based High-Level Controller (HLC) for strategic stock selection with a Deep Deterministic Policy Gradient (DDPG)-based Low-Level Controller (LLC) tasked with optimizing trade executions to enhance portfolio value. In our empirical analysis, comparing the HRT agent with standalone DRL models and the S&P 500 benchmark during both bullish and bearish market conditions, we achieve a positive and higher Sharpe ratio. This advancement not only underscores the efficacy of incorporating hierarchical structures into DRL strategies but also mitigates the aforementioned challenges, paving the way for designing more profitable and robust trading algorithms in complex markets.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics from deep reinforcement learning (PPO, DDPG) and hierarchical MDP formulations, indicating high complexity. It presents a backtest-ready empirical analysis with Sharpe ratios, comparisons to benchmarks, and visualizations on S&P 500 data, though real-world implementation details like slippage and latency are not deeply explored.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Develop robust automated trading system<br>addressing dimensionality, inertia, and diversification"] --> B["Methodology: Bi-Level HRT Architecture"]
  B --> C["Data Inputs<br>Historical Stock Market Data"]
  C --> D["High-Level Controller<br>PPO Agent: Strategic Stock Selection"]
  D --> E["Low-Level Controller<br>DDPG Agent: Execution Optimization"]
  E --> F["Computational Process<br>Bi-Level HRL Integration"]
  F --> G["Key Outcomes"]
  G --> H["Superior Sharpe Ratio vs. Benchmarks"]
  G --> I["Effective Mitigation of DRL Challenges"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes Hierarchical Reinforced Trader (HRT), a bi‑level reinforcement learning framework that separates sparse high‑level directional selection from risk‑aware low‑level execution for text‑aware portfolio management.

Source sentence
We propose Hierarchical Reinforced Trader (HRT), a bi-level reinforcement learning framework for text-aware portfolio management in multi-asset equity markets.

Assumes 5

  • Trading decisions are based on a fixed 89‑stock Nasdaq universe. market
    Source sentence
    fixed 89-stock Nasdaq universe, using 2013–2018 for training, 2019 for validation, and 2020–2023 for final out-of-sample testing
  • Text signals are pre‑computed, timestamp‑aligned sentiment and risk scores. data
    Source sentence
    text is converted into timestamp-aligned structured sentiment and risk signals
  • Turnover, drawdown and text‑risk penalties constrain the low‑level controller. execution
    Source sentence
    risk-aware Low-Level Controller (LLC) converts these directions into feasible portfolio weight adjustments under turnover, drawdown, and text-risk penalties
  • Transaction costs are modeled as a linear turnover penalty and examined under stress‑test cost levels. model
    Source sentence
    reduces daily turnover from 0.112 to 0.090, and remains robust under transaction-cost stress
Show 1 more
  • The benchmark protocol assumes no data leakage and uses a public timestamp‑clean protocol. other
    Source sentence
    public benchmark data availability under the same timestamp-clean text-aware protocol

Data

Backtest sample: 2020 – 2023 · US equities (89‑stock Nasdaq universe) · equities · daily

  • open stock‑news benchmark benchmark public
    2013–2023 · 89‑stock Nasdaq universe
    Source sentence
    We evaluate HRT on an open stock-news benchmark with a fixed 89-stock Nasdaq universe, using 2013–2018 for training, 2019 for validation, and 2020–2023 for final out-of-sample testing
  • FN‑SPID financial news dataset text-news public
    Source sentence
    benchmark constructed from FN‑SPID financial news and daily market data with precomputed LLM‑generated sentiment and risk signals

Recognised sources: FNSPID

Costs

Modeled Costs are part of the reported headline results.

Costs are incorporated via a turnover penalty and evaluated across 0–50 bps per trade in sensitivity analysis.

Source sentence
All strategies deteriorate as transaction costs increase, but HRT achieves the highest Sharpe ratio among the reported strategies across the evaluated 1–50 bps cost settings

Finds 5

Reported performance (Hierarchical Reinforced Trader (HRT)): Sharpe 1.24 (benchmark 0.80) · annual return 23.4% · total return 132.1% · max drawdown 24.5%

  • HRT improves Sharpe ratio from 1.06 (HRT‑Base) to 1.24 and reduces daily turnover from 0.112 to 0.090 in out‑of‑sample testing (2020‑2023). performance
    Source sentence
    The full model improves Sharpe from 1.06 for HRT-Base to 1.24, reduces daily turnover from 0.112 to 0.090
  • Maximum drawdown improves from –0.305 to –0.245 when moving from HRT‑Base to full HRT. performance
    Source sentence
    improving maximum drawdown from -0.305 to -0.245
  • Ablation shows that adding sparse high‑level activation, text‑risk signal, and risk‑aware LLC sequentially improves Sharpe and reduces turnover and drawdown. performance
    Source sentence
    Sparse high-level activation sacrifices some raw return but tends to improve Sharpe, drawdown, and turnover. Adding text-derived risk information reduces ex-post text-risk exposure and improves path-level downside control. The full risk-aware LLC further improves the return–risk–cost balance
  • Under transaction‑cost stress tests (0–50 bps), HRT maintains the highest Sharpe ratio among all evaluated strategies. performance
    Source sentence
    HRT achieves the highest Sharpe ratio among the reported strategies across the evaluated 1–50 bps cost settings
  • The hierarchical design yields a stronger overall return‑risk‑cost trade‑off compared to flat RL, alpha‑only selection, and non‑RL baselines. performance
    Source sentence
    HRT delivers the strongest learning-based return–risk–cost trade-off

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 23 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (23)

  1. FinRL-DeepSeek : LLM -Infused Risk-Sensitive Reinforcement Learning for Trading Agents — → 2502.07393
  2. Dynamic Programming and Optimal Control: Volume I (2012)
  3. FNSPID : A Comprehensive Financial News Dataset in Time Series — → 2402.06698
  4. Addressing Function Approximation Error in Actor-Critic Methods (2018)
  5. Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning — → 2301.10724
  6. LightGBM : A Highly Efficient Gradient Boosting Decision Tree (2017)
  7. Market Sentiment-Aware Deep Reinforcement Learning Approach for Stock Portfolio Allocation (2021)
  8. Continuous Control with Deep Reinforcement Learning (2015)
and 15 more
  1. Practical Deep Reinforcement Learning Approach for Stock Trading (2018)
  2. FinRL : Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021)
  3. FinRL-Meta : Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022)
  4. Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2024) — link
  5. Portfolio Selection (1952)
  6. Asynchronous Methods for Deep Reinforcement Learning (2016)
  7. Sentiment and Knowledge Based Algorithmic Trading with Deep Reinforcement Learning (2022)
  8. Hierarchical Reinforcement Learning: A Comprehensive Survey (2021)
  9. EarnHFT : Efficient Hierarchical Reinforcement Learning for High Frequency Trading — → 2309.12891
  10. Proximal Policy Optimization Algorithms (2017)
  11. Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution (2020)
  12. Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (2020)
  13. Qlib : An AI-Oriented Quantitative Investment Platform (2020)
  14. Instruct-FinGPT : Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
  15. A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485

6 of 23 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.