Reinforcement Learning for Speculative Trading under Exploratory Framework

Paper: arXiv 2604.02035

Authors: Yun Zhao, Alex S. L. Tse, Harry Zheng

Abstract

We study a speculative trading problem within the exploratory reinforcement learning (RL) framework of Wang et al. [2020]. The problem is formulated as a sequential optimal stopping problem over entry and exit times under general utility function and price process. We first consider a relaxed version of the problem in which the stopping times are modeled by the jump times of Cox processes driven by bounded, non-randomized intensity controls. Under the exploratory formulation, the agent’s randomized control is characterized via the probability measure over the jump intensities, and their objective function is regularized by Shannon’s differential entropy. This yields a system of the exploratory HJB equations and Gibbs distributions in closed-form as the optimal policy. Error estimates and convergence of the RL objective to the value function of the original problem are established. Finally, an RL algorithm is designed, and its implementation is showcased in a pairs-trading application.

Complexity vs Empirical Score

• Math Complexity: 8.5/10
• Empirical Rigor: 5.5/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a rigorous theoretical extension of exploratory RL to speculative optimal stopping using Cox processes, differential entropy, and exploratory HJB equations. While the mathematical derivations and error estimates are exceptionally strong, the empirical component is limited to a single pairs-trading application, making the backtest secondary to the analytical contributions.

Research Flowchart

  flowchart TD
    A[Goal: Speculative Trading via Exploratory RL & Optimal Stopping] --> B[Formulate Problem: Entry/Exit as Cox Process Jump Times]
    B --> C[Inputs: Asset Price Process, Utility Function & Shannon Entropy]
    C --> D[Derive Exploratory HJB Equations & Closed-Form Gibbs Policy]
    D --> E[Prove Error Estimates & Convergence to Original Value Function]
    E --> F[Algorithm Design & Pairs-Trading Implementation]
    F --> G[Outcomes: Robust RL Framework for Continuous-Time Trading]