Paper: arXiv 2311.05743

Authors: Gang Hu

Abstract

This study enhances a Deep Q-Network (DQN) trading model by incorporating advanced techniques like Prioritized Experience Replay, Regularized Q-Learning, Noisy Networks, Dueling, and Double DQN. Extensive tests on assets like BTC/USD and AAPL demonstrate superior performance compared to the original model, with marked increases in returns and Sharpe Ratio, indicating improved risk-adjusted rewards. Notably, convolutional neural network (CNN) architectures, both 1D and 2D, significantly boost returns, suggesting their effectiveness in market trend analysis. Across instruments, these enhancements have yielded stable and high gains, eclipsing the baseline and highlighting the potential of CNNs in trading systems. The study suggests that applying sophisticated deep learning within reinforcement learning can greatly enhance automated trading, urging further exploration into advanced methods for broader financial applicability. The findings advocate for the continued evolution of AI in finance.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 9.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced deep reinforcement learning techniques (Prioritized Experience Replay, Dueling/Double DQN, Noisy Networks, CNN architectures) with explicit mathematical formulations, earning a high complexity score. It demonstrates high empirical rigor through extensive backtesting on real financial data (BTC/USD and AAPL), reporting specific metrics like returns and Sharpe Ratios, and comparing performance against baselines.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Enhance DQN Trading Model"] --> B["Methodology:<br>Advanced Techniques"]
  B --> C["Data & Inputs:<br>Historical Market Data<br>BTC/USD & AAPL"]
  C --> D["Computational Process:<br>CNN Architectures<br>1D & 2D Integration"]
  D --> E["Key Findings & Outcomes:<br>Superior Performance<br>Higher Returns & Sharpe Ratio"]
  
  subgraph B
      B1["Prioritized Experience Replay"]
      B2["Regularized Q-Learning"]
      B3["Noisy Networks & Dueling"]
      B4["Double DQN"]
  end

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes an enhanced Deep Q‑Network trading model that incorporates Prioritized Experience Replay, Regularized Q‑Learning, Noisy Networks, Dueling DQN, and Double DQN.

Source sentence
This study enhances a Deep Q-Network (DQN) trading model by incorporating advanced techniques like Prioritized Experience Replay, Regularized Q-Learning, Noisy Networks, Dueling, and Double DQN.

Assumes 2

  • The market data are represented as daily candlestick charts. data
    Source sentence
    All datasets were formatted into daily candlestick charts.
  • Benchmarking uses the same datasets as prior work [8]. data
    Source sentence
    For benchmarking the model’s effectiveness, the same datasets as those employed by [8] were utilized

Data

  • GOOGL · Yahoo Finance / Google Finance market-prices public
    2010/01/01 – 2020/08/24
    Source sentence
    GOOGL 2010/01/01 2018/01/01 2020/08/24
  • AAPL · Yahoo Finance / Google Finance market-prices public
    2010/01/01 – 2020/08/24
    Source sentence
    AAPL 2010/01/01 2018/01/01 2020/08/24
  • KSS · Yahoo Finance / Google Finance market-prices public
    1999/01/01 – 2020/08/24
    Source sentence
    KSS 1999/01/01 2018/01/01 2020/08/24
  • BTC-USD · Yahoo Finance / Google Finance crypto-onchain public
    2014/09/17 – 2020/08/26
    Source sentence
    BTC-USD 2014/09/17 2018/01/01 2020/08/26

Costs

Unclear The classifier was not confident.

Finds 4

  • The DQN‑windowed variant achieved a Total Return of 737% and Sharpe Ratio of 0.195, the highest among all models on AAPL. performance
    Source sentence
    Our DQN-windowed variant achieved an outstanding Total Return of 737%, surpassing all other models, accompanied by the highest Sharpe Ratio of 0.195 in this cohort.
  • The improved DQN‑vanilla model yielded a Total Return of 517% and Sharpe Ratio of 0.181, surpassing the prior DQN‑vanilla implementation. performance
    Source sentence
    The improved DQN-vanilla model’s performance was particularly remarkable, yielding a Total Return of 517% and a Sharpe Ratio of 0.181
  • On BTC/USD, CNN2D achieved returns of 3897% and Sharpe Ratio improvement to 0.085, far outstripping prior benchmarks. performance
    Source sentence
    the CNN1D and CNN2D models, through the introduction of convolutional layers
  • The DQN‑vanilla variant on BTC/USD improved arithmetic return from 261% to 287% and Sharpe Ratio from 0.076 to 0.085. performance
    Source sentence
    the DQN-vanilla variant exhibited an outstanding arithmetic return of 287% compared to the original’s 261%, and a Sharpe Ratio improvement from 0.076 to 0.085

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 33 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (25)

  1. Quantitative Trading: How to Build Your Own Algorithmic Trading Business (2021)
  2. High Frequency Trading (2015)
  3. Deep Learning for Portfolio Optimization (2020) — link
  4. Learning Financial Asset-Specific Trading Rules via Deep Reinforcement Learning (2022)
  5. Using Genetic Algorithms to Find Technical Trading Rules (1999)
  6. Generating Trading Rules on the Stock Markets with Genetic Programming (2004)
  7. Mining Associative Classification Rules with Stock Trading Data–A GA-Based Method (2010)
  8. A Reinforcement Learning Based Encoder-Decoder Framework for Learning Stock Trading Rules (2021)
and 17 more
  1. High-dimensional stock portfolio trading with deep reinforcement learning (2022)
  2. Deep reinforcement learning with double q-learning (2016)
  3. Dueling network architectures for deep reinforcement learning (2016)
  4. Noisy networks for exploration (2017) — link
  5. Prioritized experience replay (2015) — link
  6. Human-level control through deep reinforcement learning (2015)
  7. Modeling purposeful adaptive behavior with the principle of maximum causal entropy (2010)
  8. first_page settings Order Article Reprints Open AccessArticle Pore-Level Multiphase Simulations of Realistic Distillation Membranes for Water Desalination (2022)
  9. Perspectives on predictive power of multimodal deep learning: Surprises and future directions (2018)
  10. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care (2018)
  11. Financial trading as a game: A deep reinforcement learning approach (2018) — link
  12. An application of deep reinforcement learning to algorithmic trading (2021)
  13. Application of deep reinforcement learning in stock trading strategies and stock forecasting (2020)
  14. Regularization in reinforcement learning (2011)
  15. Practical deep reinforcement learning approach for stock trading (2018) — link
  16. Adaptive stock trading strategies with deep reinforcement learning methods (2020)
  17. Deep Q-trading. cslt. riit. tsinghua. edu. cn (2017)

0 of 25 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.