Exploratory Optimal Stopping: A Singular Control Formulation

Paper: arXiv 2408.09335

Authors: Jodi Dianetti, Giorgio Ferrari, Renyuan Xu

Abstract

This paper explores continuous-time and state-space optimal stopping problems from a reinforcement learning perspective. We begin by formulating the stopping problem using randomized stopping times, where the decision maker’s control is represented by the probability of stopping within a given time-specifically, a bounded, non-decreasing, càdlàg control process. To encourage exploration and facilitate learning, we introduce a regularized version of the problem by penalizing the performance criterion with the cumulative residual entropy of the randomized stopping time. The regularized problem takes the form of an (n+1)-dimensional degenerate singular stochastic control with finite-fuel, where the regularized free boundary becomes the graph of a function mapping the state variable of the original stopping problem into the probability of stopping. We address this singular control problem through the dynamic programming principle, which enables us to identify the unique optimal exploratory strategy. Finally, we propose both model-based and model-free reinforcement learning algorithms tailored for exploratory optimal stopping problems. We establish policy improvement guarantees for the proposed algorithms. Moreover, the model-free method is of actor-critic type and it is scalable in high-dimensions under neural network parameterization.

Complexity vs Empirical Score

• Math Complexity: 9.5/10
• Empirical Rigor: 3.5/10
• Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: This paper presents a rigorous theoretical formulation of exploratory optimal stopping via singular stochastic control with cumulative residual entropy regularization. While it establishes deep mathematical properties, viscosity solutions, and theoretical policy convergence, the empirical validation relies primarily on standard synthetic benchmarks rather than real-world quantitative trading data.

Research Flowchart

  flowchart TD
    A["Research Goal: Formulate Continuous-Time<br>Optimal Stopping for Reinforcement Learning"] --> B["Method: Introduce Randomized Stopping Times<br>& Cumulative Residual Entropy Penalty"]
    B --> C["Formulation: Transform to (n+1)-Dim Degenerate<br>Singular Stochastic Control Problem"]
    C --> D["Analysis: Apply Dynamic Programming Principle<br>to derive Free Boundary & Optimal Strategy"]
    D --> E["Algorithm Design: Develop Model-Based &<br>Model-Free Actor-Critic RL Methods"]
    E --> F["Outcomes: Proven Policy Improvement Guarantees<br>& Scalable High-Dim Neural Network Solver"]

Cited References (405)

1. Convergence of probability measures (1999)
2. Exploratory optimal stopping: A singular control formulation — → 2408.09335
3. Exploratory LQG mean field games with entropy regularization (2022)
4. Denseness of adapted processes among causal couplings (2018) — link
5. Exploratory optimal stopping: A singular control formulation — → 2408.09335
6. Ergodic mean-field games of singular control with regime-switching (extended version) (2023) — link
7. Strong solutions of SDE s with singular drift and S obolev diffusion coefficients (2005)
8. Minimal solutions of master equations for extended mean field games (2024)

5 of 405 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.