Paper: arXiv 2309.12891

Authors: Molei Qin, Shuo Sun, Wentao Zhang, Haochong Xia, Xinrun Wang, Bo An

Abstract

High-frequency trading (HFT) uses computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performance. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second-level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematical concepts like dynamic programming, KL divergence, and hierarchical Markov Decision Processes, placing it in the high complexity range. It demonstrates strong empirical rigor through extensive experiments in a high-fidelity simulation environment, evaluating against multiple baselines and financial criteria, making it highly backtest-ready.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Develop efficient RL for HFT<br>in volatile Crypto markets"] --> B["Data: BTC/ETH Market Data<br>High-frequency time series"]

  B --> C["Stage 1: Q-Teacher Pre-training<br>Dynamic Programming for<br>Optimal Action Values"]

  C --> D["Stage 2: Agent Pool Creation<br>Train 100s of RL agents<br>Filter by profitability & return preference"]

  D --> E["Stage 3: Minute-Level Router<br>Dynamic Agent Selection<br>for stable trend adaptation"]

  E --> F["Simulated Trading Environment<br>High-fidelity backtesting"]

  F --> G["Key Outcomes<br>1. 30% higher profitability than SOTA<br>2. Outperforms 6 baselines<br>3. Robust across market trends"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes EarnHFT, a novel three-stage hierarchical reinforcement learning framework for high‑frequency cryptocurrency trading.

Source sentence
we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT.

Assumes 2

  • Only long positions are allowed (position must be non‑negative). execution
    Source sentence
    Position at time t is denoted as Pt and Pt ≥0, indicating only a long position is permitted in this formulation.
  • A fixed commission fee rate is applied to each trade (0 for BTCT, 0.02% for other datasets). market
    Source sentence
    For the trading setting, the commission fee rate is 0 for BTCT and 0.02% for the remaining datasets following the policy of Binance.

Data

  • BTC/TUSD market-prices unknown
    23/03/30 to 23/05/15
    Source sentence
    BTC/TUSD Sideways 4057140 23/03/30 23/05/15
  • BTC/USDT market-prices unknown
    22/09/01 to 22/10/15
    Source sentence
    BTC/USDT Sideways 3884400 22/09/01 22/10/15
  • ETH/USDT market-prices unknown
    22/05/01 to 22/06/15
    Source sentence
    ETH/USDT Bear 3970800 22/05/01 22/06/15
  • GALA/USDT market-prices unknown
    22/07/01 to 22/08/15
    Source sentence
    GALA/USDT Bull 3970740 22/07/01 22/08/15

Costs

Modeled Costs are part of the reported headline results.

Commission fee rate is 0 for BTCT and 0.02% for the other three crypto datasets.

Source sentence
For the trading setting, the commission fee rate is 0 for BTCT and 0.02% for the remaining datasets following the policy of Binance.

Finds 2

  • EarnHFT significantly outperforms six state‑of‑the‑art baselines on six financial criteria, beating the runner‑up by 30% in profitability. performance
    Source sentence
    EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.
  • EarnHFT achieves the highest profit in all four datasets and the highest risk‑adjusted profit in three datasets. performance
    Source sentence
    our method achieves the highest profit in all 4 datasets and the highest risk-adjusted profit in 3 datasets.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 6 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 17 papers in the archive (13 by independent authors): HedgeAgents: A Balanced-aware Multi-agent Financial Trading … (uses its method or data)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.