Paper: arXiv 2309.12891
Authors: Molei Qin, Shuo Sun, Wentao Zhang, Haochong Xia, Xinrun Wang, Bo An
Abstract
High-frequency trading (HFT) uses computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performance. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second-level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs advanced mathematical concepts like dynamic programming, KL divergence, and hierarchical Markov Decision Processes, placing it in the high complexity range. It demonstrates strong empirical rigor through extensive experiments in a high-fidelity simulation environment, evaluating against multiple baselines and financial criteria, making it highly backtest-ready.
Research Flowchart
flowchart TD A["Research Goal<br>Develop efficient RL for HFT<br>in volatile Crypto markets"] --> B["Data: BTC/ETH Market Data<br>High-frequency time series"] B --> C["Stage 1: Q-Teacher Pre-training<br>Dynamic Programming for<br>Optimal Action Values"] C --> D["Stage 2: Agent Pool Creation<br>Train 100s of RL agents<br>Filter by profitability & return preference"] D --> E["Stage 3: Minute-Level Router<br>Dynamic Agent Selection<br>for stable trend adaptation"] E --> F["Simulated Trading Environment<br>High-fidelity backtesting"] F --> G["Key Outcomes<br>1. 30% higher profitability than SOTA<br>2. Outperforms 6 baselines<br>3. Robust across market trends"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes EarnHFT, a novel three-stage hierarchical reinforcement learning framework for high‑frequency cryptocurrency trading.
Source sentence
we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT.
Assumes 2
- Only long positions are allowed (position must be non‑negative). execution
Source sentence
Position at time t is denoted as Pt and Pt ≥0, indicating only a long position is permitted in this formulation.
- A fixed commission fee rate is applied to each trade (0 for BTCT, 0.02% for other datasets). market
Source sentence
For the trading setting, the commission fee rate is 0 for BTCT and 0.02% for the remaining datasets following the policy of Binance.
Data
- BTC/TUSD market-prices unknown
Source sentence
BTC/TUSD Sideways 4057140 23/03/30 23/05/15
- BTC/USDT market-prices unknown
Source sentence
BTC/USDT Sideways 3884400 22/09/01 22/10/15
- ETH/USDT market-prices unknown
Source sentence
ETH/USDT Bear 3970800 22/05/01 22/06/15
- GALA/USDT market-prices unknown
Source sentence
GALA/USDT Bull 3970740 22/07/01 22/08/15
Costs
Modeled
Commission fee rate is 0 for BTCT and 0.02% for the other three crypto datasets.
Source sentence
For the trading setting, the commission fee rate is 0 for BTCT and 0.02% for the remaining datasets following the policy of Binance.
Finds 2
- EarnHFT significantly outperforms six state‑of‑the‑art baselines on six financial criteria, beating the runner‑up by 30% in profitability. performance
Source sentence
EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.
- EarnHFT achieves the highest profit in all four datasets and the highest risk‑adjusted profit in three datasets. performance
Source sentence
our method achieves the highest profit in all 4 datasets and the highest risk-adjusted profit in 3 datasets.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 6 related papers). Coverage is partial, so this is neither replication nor refutation.
17 papers in the archive : HedgeAgents: A Balanced-aware Multi-agent Financial Trading …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.