Paper: arXiv 2510.10260

Authors: Junyan Ye, Hoi Ying Wong, Kyunghyun Park

Abstract

We propose and analyze a continuous-time robust reinforcement learning framework for optimal stopping under ambiguity. In this framework, an agent chooses a robust exploratory stopping time motivated by two objectives: robust decision-making under ambiguity and learning about the unknown environment. Here, ambiguity refers to considering multiple probability measures dominated by a reference measure, reflecting the agent’s awareness that the reference measure representing her learned belief about the environment would be erroneous. Using the $g$-expectation framework, we reformulate the optimal stopping problem under ambiguity as a robust exploratory control problem with Bernoulli distributed controls. We then characterize the optimal Bernoulli distributed control via backward stochastic differential equations and, based on this, construct the robust exploratory stopping time that approximates the optimal stopping time under ambiguity. Last, we establish a policy iteration theorem and implement it as a reinforcement learning algorithm. Numerical experiments demonstrate the convergence, robustness, and scalability of our reinforcement learning algorithm across different levels of ambiguity and exploration.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 5.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper provides a highly rigorous theoretical foundation for robust reinforcement learning under model ambiguity using continuous-time BSDEs and g-expectations. While its empirical validation relies on synthetic financial scenarios (American option valuation) rather than historical market data backtests, the theoretical proofs, algorithm design, and scalability analyses are very strong.

Research Flowchart

  flowchart TD
    A["Research Goal:<br>Continuous-Time Robust RL for<br>Optimal Stopping under Ambiguity"] --> B["Input / Baseline:<br>Reference measure (learned belief)<br>& g-Expectation framework"]
    
    B --> C["Reformulation:<br>Optimal stopping mapped to<br>robust exploratory control with<br>Bernoulli distributed controls"]
    
    C --> D["Theoretical Characterization:<br>Solve via Backward Stochastic<br>Differential Equations (BSDEs)"]
    
    D --> E["Algorithm Design:<br>Prove Policy Iteration Theorem<br>& implement Robust RL algorithm"]
    
    E --> F["Outcomes & Experiments:<br>Guaranteed convergence, robustness to<br>model misspecification, & scalability"]

Cited References (95)

  1. Sensitivity of robust optimization problems under drift and volatility uncertainty — → 2311.11248
  2. Deep Splitting Method for Parabolic PDEs (2021)
  3. An overview on deep learning-based approximation methods for partial differential equations (2020) — link
  4. An extension of Skorohod’s almost sure representation theorem (1983)
  5. Learning an Optimal Investment Policy with Transaction Costs via a Randomized Dynkin Game (2024) — link
  6. Filtration-consistent nonlinear expectations and related $g$ -expectations (2002)
  7. Intensity-based framework and penalty formulation of optimal stopping problems (2007)
  8. Learning to Optimally Stop Diffusion Processes, with Financial Applications (2026) — link
and 87 more
  1. Randomized Optimal Stopping Problem in Continuous Time and Reinforcement Learning Algorithm (2024)
  2. Reflected solutions of backward SDE , and related obstacle problems for PDEs (1997)
  3. Backward stochastic differential equations in finance (1997)
  4. Optimal consumption with Hindy–Huang–Kreps preferences under nonlinear expectations (2022)
  5. Quadratic Convergence for Valuing American Options Using a Penalty Method (2002)
  6. Approximation Error Analysis of Some Deep Backward Schemes for Nonlinear PDEs (2022)
  7. Deep learning (2016)
  8. Solving high-dimensional partial differential equations using deep learning (2018)
  9. Deep backward schemes for high-dimensional nonlinear PDEs (2020)
  10. Chapter Eight. BSDEs And Applications (2009)
  11. Penalization method for reflected backward stochastic differential equations with one r.c.l.l. barrier (2005)
  12. Stochastic differential equations and applications (2007)
  13. Full error analysis of the random deep splitting method for nonlinear parabolic PDEs and PIDEs — → 2405.05192
  14. Backward stochastic differential equations and quasilinear parabolic partial differential equations (2005)
  15. Adapted solution of a backward stochastic differential equation (1990)
  16. Backward SDE and related $g$-expectation (1997)
  17. The smallest $g$ -supermartingale and reflected BSDE with single and double $L^2$ obstacles (2005)
  18. DGM : A deep learning algorithm for solving partial differential equations (2018)
  19. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  20. A Comprehensive Survey on Safe Reinforcement Learning (2015)
  21. Robust Adversarial Reinforcement Learning (2017)
  22. Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model (2016) — link
  23. Optimal stopping and divestment timing under scenario ambiguity and learning (2025) — link
  24. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  25. Backward Stochastic Differential Equations (2017)
  26. Reinforcement Learning: An Introduction (1998)
  27. Mastering the game of Go with deep neural networks and tree search (2016)
  28. Mastering the game of Go without human knowledge (2017)
  29. Human-level control through deep reinforcement learning (2015)
  30. q-Learning in Continuous Time (2023)
  31. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  32. End-to-end training of deep visuomotor policies (2016)
  33. Bayes and minimax solutions of sequential decision problems (1949)
  34. Optimum character of the sequential probability ratio test (1948)
  35. Dusenberry’s ratcheting of consumption: optimal dynamic consumption and investment given intolerance for any decline in standard of living (1995)
  36. Lifetime consumption and investment: retirement and constrained borrowing (2010)
  37. Optimal stopping and free-boundary problems (2006)
  38. Optimal stopping rules (2008)
  39. Ambiguity, risk, and asset returns in continuous time (2002)
  40. Recursive multiple-priors (2003)
  41. Limit laws for non-additive probabilities and their frequentist interpretation (1999)
  42. Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms (2022)
  43. Exploratory HJB Equations and Their Convergence (2022)
  44. Choquet regularization for continuous-time reinforcement learning (2023)
  45. Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market (2024)
  46. Learning equilibrium mean-variance strategy (2023)
  47. Learning Merton’s Strategies in an Incomplete Market: Recursive Entropy Regularization and Biased Gaussian Exploration (2023) — link
  48. On the multidimensional controller-and-stopper games (2013)
  49. Optimal stopping for non-linear expectations— Part I (2011)
  50. Optimal stopping for non-linear expectations— Part II (2011)
  51. OPTIMAL STOPPING UNDER ADVERSE NONLINEAR EXPECTATION AND RELATED GAMES (2015)
  52. Robust Retirement with Return Ambiguity: Optimal $G$ -Stopping Time in Dual Space (2023)
  53. Robust retirement and life insurance with inflation risk and model ambiguity (2023)
  54. Optimal stopping with multiple priors (2009)
  55. Numerical method for nonlinear Kolmogorov PDEs via sensitivity analysis (2026+) — link
  56. Optimal dividends under model uncertainty (2023)
  57. Optimal ergodic harvesting under ambiguity (2022)
  58. A Knightian irreversible investment problem (2022)
  59. Irreversible Consumption Habit under Ambiguity: Singular Control and Optimal $G$ -Stopping Time (2025)
  60. Robust dividend policy: Equivalence of Epstein-Zin and Maenhout preferences — → 2406.12305
  61. A control approach to robust utility maximization with logarithmic utility and time-consistent penalties (2007)
  62. Ambiguity aversion, robustness, and the variational representation of preferences (2006)
  63. Robust consumption-investment with return ambiguity: A dual approach with volatility ambiguity (2022)
  64. Optimal investments for risk-and ambiguity-averse preferences: a duality approach (2007)
  65. A smooth model of decision making under ambiguity (2005)
  66. An Entropy Regularized BSDE Approach to Bermudan Options and Games (2025) — link
  67. Deep optimal stopping (2019)
  68. Solving high-dimensional optimal stopping problems using deep learning (2021)
  69. Neural optimal stopping boundary (2025)
  70. Optimal stopping via randomized neural networks (2024)
  71. Deep penalty methods: A class of deep learning algorithms for solving high dimensional optimal stopping problems — → 2405.11392
  72. Convergence Analysis of the Deep Splitting Scheme: The Case of Partial Integro-Differential Equations and the Associated Forward Backward SDEs with Jumps (2025)
  73. Stochastic differential equations (2003)
  74. Reinforcement learning under model mismatch (2017)
  75. Robust deep reinforcement learning against adversarial perturbations on state observations (2020)
  76. Reinforcement Learning in Robust Markov Decision Processes (2016)
  77. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning (2025)
  78. Stochastic Integration and Differential Equations (2005)
  79. Limit theorems for stochastic processes (2013)
  80. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage (2023)
  81. Robust reinforcement learning using offline data (2022)
  82. Robust reinforcement learning (2005)
  83. Exploratory optimal stopping: A singular control formulation — → 2408.09335
  84. Convergence of policy iteration for entropy-regularized stochastic control problems (2025)
  85. Group sequential methods in the design and analysis of clinical trials (1977)
  86. Multi-armed bandit allocation indices (2011)
  87. A survey of maintenance models: the control and surveillance of deteriorating systems (1976)

5 of 95 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.