Learning to Optimally Stop Diffusion Processes, with Financial Applications

Paper: arXiv 2408.09242

Authors: Min Dai, Yu Sun, Zuo Quan Xu, Xun Yu Zhou

Abstract

We study optimal stopping for diffusion processes with unknown model primitives within the continuous-time reinforcement learning (RL) framework developed by Wang et al. (2020), and present applications to option pricing and portfolio choice. By penalizing the corresponding variational inequality formulation, we transform the stopping problem into a stochastic optimal control problem with two actions. We then randomize controls into Bernoulli distributions and add an entropy regularizer to encourage exploration. We derive a semi-analytical optimal Bernoulli distribution, based on which we devise RL algorithms using the martingale approach established in Jia and Zhou (2022a). We establish a policy improvement theorem and prove the fast convergence of the resulting policy iterations. We demonstrate the effectiveness of the algorithms in pricing finite-horizon American put options, solving Merton’s problem with transaction costs, and scaling to high-dimensional optimal stopping problems. In particular, we show that both the offline and online algorithms achieve high accuracy in learning the value functions and characterizing the associated free boundaries.

Complexity vs Empirical Score

• Math Complexity: 8.5/10
• Empirical Rigor: 6.5/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a novel continuous-time RL framework for optimal stopping by converting the problem into a stochastic control setting using penalty methods and Bernoulli distribution randomization. It delivers strong mathematical proofs for policy improvement and fast convergence, complemented by practical simulation applications to option pricing and high-dimensional settings.

Research Flowchart

  flowchart TD
    A[Goal: Optimal Stopping for Unknown Diffusion Processes in RL] --> B[Transform Problem: Penalize Variational Inequality to Stochastic Control]
    B --> C[Methodology: Randomize to Bernoulli Actions + Add Entropy Regularization]
    C --> D[Algorithm Design: Derive Semi-Analytical Policy via Martingale Approach]
    D --> E[Inputs & Applications: American Options, Merton Problem, High-Dim Stopping]
    E --> F[Findings: Fast Policy Convergence & Accurate Free Boundary Learning]