Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets

Paper: arXiv 2301.08688

Authors: Peer Nagy, Jan-Peter Calliess, Stefan Zohren

Abstract

We employ deep reinforcement learning (RL) to train an agent to successfully translate a high-frequency trading signal into a trading strategy that places individual limit orders. Based on the ABIDES limit order book simulator, we build a reinforcement learning OpenAI gym environment and utilise it to simulate a realistic trading environment for NASDAQ equities based on historic order book messages. To train a trading agent that learns to maximise its trading return in this environment, we use Deep Duelling Double Q-learning with the APEX (asynchronous prioritised experience replay) architecture. The agent observes the current limit order book state, its recent history, and a short-term directional forecast. To investigate the performance of RL for adaptive trading independently from a concrete forecasting algorithm, we study the performance of our approach utilising synthetic alpha signals obtained by perturbing forward-looking returns with varying levels of noise. Here, we find that the RL agent learns an effective trading strategy for inventory management and order placing that outperforms a heuristic benchmark trading strategy having access to the same signal.

Complexity vs Empirical Score

• Math Complexity: 6.5/10
• Empirical Rigor: 8.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a sophisticated application of advanced deep reinforcement learning (ApeX D3QN) to high-frequency execution in limit order book markets using ABIDES. The empirical framework is rigorous, isolating execution quality via controlled synthetic alpha signals, though detailed mathematical proofs are absent.

Research Flowchart

  flowchart TD
    A[Research Goal: Train DRL Agent for Optimal LOB Order Execution] --> B[Environment: ABIDES OpenAI Gym with Historic NASDAQ LOB Data]
    B --> C[Inputs: LOB State, History, and Synthetic Alpha Signal]
    C --> D[Model: Asynchronous Deep Duelling Double Q-Learning]
    D --> E[Training: APEX Architecture with Prioritised Replay]
    E --> F[Findings: Outperforms Heuristic Benchmark in Inventory & Execution]