Paper: arXiv 2405.16449

Authors: Xuefeng Gao, Lingfei Li, Xun Yu Zhou

Abstract

We study continuous-time reinforcement learning (RL) for stochastic control in which system dynamics are governed by jump-diffusion processes. We formulate an entropy-regularized exploratory control problem with stochastic policies to capture the exploration–exploitation balance essential for RL. Unlike the pure diffusion case initially studied by Wang et al. (2020), the derivation of the exploratory dynamics under jump-diffusions calls for a careful formulation of the jump part. Through a theoretical analysis, we find that one can simply use the same policy evaluation and $q$-learning algorithms in Jia and Zhou (2022a, 2023), originally developed for controlled diffusions, without needing to check a priori whether the underlying data come from a pure diffusion or a jump-diffusion. However, we show that the presence of jumps ought to affect parameterizations of actors and critics in general. We investigate as an application the mean–variance portfolio selection problem with stock price modelled as a jump-diffusion, and show that both RL algorithms and parameterizations are invariant with respect to jumps. Finally, we present a detailed study on applying the general theory to option hedging.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper provides a rigorous mathematical framework extending continuous-time reinforcement learning to jump-diffusion processes, deriving theoretical well-posedness and algorithmic invariance results. While it offers solid theoretical actor-critic derivations for financial applications like portfolio selection and option hedging, it focuses on mathematical theory and simulation design rather than extensive empirical backtesting on historical data.

Research Flowchart

  flowchart TD
    A[Research Goal: Continuous-Time RL for<br>Jump-Diffusion Stochastic Control] --> B[Formulate Entropy-Regularized<br>Exploratory Control Problem]
    B --> C[Theoretical Analysis of<br>Policy Evaluation & Q-Learning]
    C --> D[Financial Applications:<br>Mean-Variance Portfolio & Option Hedging]
    D --> E[Outcome 1: Pure Diffusion RL Algorithms<br>Apply to Jump-Diffusions Without Prior Knowledge]
    D --> F[Outcome 2: Jumps Impact Actor-Critic Parameterization,<br>except in Mean-Variance Selection]