Paper: arXiv 2511.17304

Authors: Jian’an Zhang

Abstract

We study reinforcement learning (RL) on volatility surfaces through the lens of Scientific AI. We ask whether axiomatic no-arbitrage laws, imposed as soft penalties on a learned world model, can reliably align high-capacity RL agents, or mainly create Goodhart-style incentives to exploit model errors. From classical static no-arbitrage conditions we build a finite-dimensional convex volatility law manifold of admissible total-variance surfaces, together with a metric law-penalty functional and a Graceful Failure Index (GFI) that normalizes law degradation under shocks. A synthetic generator produces law-consistent trajectories, while a recurrent neural world model trained without law regularization exhibits structured off-manifold errors. On this testbed we define a Goodhart decomposition (r = r^{"\mathcal{M"}} + r^\perp), where (r^\perp) is ghost arbitrage from off-manifold prediction error. We prove a ghost-arbitrage incentive theorem for PPO-type agents, a law-strength trade-off theorem showing that stronger penalties eventually worsen P&L, and a no-free-lunch theorem: under a law-consistent world model and law-aligned strategy class, unconstrained law-seeking RL cannot Pareto-dominate structural baselines on P&L, penalties, and GFI. In experiments on an SPX/VIX-like world model, simple structural strategies form the empirical law-strength frontier, while all law-seeking RL variants underperform and move into high-penalty, high-GFI regions. Volatility thus provides a concrete case where reward shaping with verifiable penalties is insufficient for robust law alignment.

Complexity vs Empirical Score

  • Math Complexity: 9.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper is dense with advanced theoretical results, including proofs of incentive and trade-off theorems, convex manifolds, and Goodhart decomposition, indicating very high mathematical complexity. Empirically, while it uses a synthetic testbed and specific metrics (e.g., GFI), the work is primarily theoretical and diagnostic with no real-world data, backtests, or code, placing it lower on empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal"] --> B["Data: Law-Consistent Trajectories"]
  B --> C["Methodology: RL on World Model"]
  C --> D["Computations"]
  subgraph D
      D1["Train RNN World Model"]
      D2["Apply Law Penalty"]
      D3["Run PPO Agent"]
  end
  D --> E{"Outcomes"}
  E --> F["Ghost Arbitrage Incentive"]
  E --> G["Law-Strength Trade-off"]
  E --> H["No Free Lunch on Volatility Frontier"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a finite‑dimensional convex volatility law manifold with a metric‑based law‑penalty functional and a Graceful Failure Index, and introduces a Goodhart decomposition of reward into on‑manifold and ghost arbitrage components.

Source sentence
we construct a finite-dimensional convex volatility law manifold of admissible total-variance surfaces, together with a metric-based law-penalty functional and a domain-agnostic Graceful Failure Index (GFI) that normalizes law degradation under shocks.

Assumes 5

  • The structural class S is non‑empty, convex, and law‑consistent with bounded law penalties and GFI. model
    Source sentence
    Assumption 3 (Structural class and world-model properties). We assume: 1. Law-consistent structural class. The structural class S ⊂Π is non-empty, convex, and law-consistent in the sense that for all s ∈S, E[Lϕ(s)] ≤LSmax and GFI(s) ≤GFISmax
  • The unconstrained policy class Π is rich enough to strictly contain S and to reach off‑manifold regions. model
    Source sentence
    Rich unconstrained class. The unconstrained policy class Π is rich enough to strictly contain S and to reach off-manifold regions
  • The synthetic generator G⋆ produces trajectories that are exactly law‑consistent (each wt lies in Mvol). data
    Source sentence
    We call G⋆ a law-consistent generator if the trajectory (wt) it induces satisfies this condition for M = Mvol.
  • The recurrent neural world model is trained without any law‑penalty regularization, using only predictive MSE loss. model
    Source sentence
    Crucially, no law penalties are used when training the world model: the loss is purely predictive.
Show 1 more
  • Shock regime changes the distribution of trajectories (via long‑variance and spot‑variance multipliers) while keeping the underlying axioms unchanged. market
    Source sentence
    To probe robustness and graceful failure, we introduce a shock regime in which the same axioms hold, but the volatility regime is stressed.

Data

  • Law‑consistent synthetic generator (G⋆) synthetic unknown
    Source sentence
    A synthetic generator produces trajectories that are exactly law-consistent, while a recurrent neural world model is trained without law regularization and therefore predicts surfaces that deviate from the law manifold in structured ways.

Costs

Not modeled A trading or execution result is reported without costs.

Finds 5

  • Ghost‑arbitrage incentive theorem shows naive PPO‑type RL is driven to increase off‑manifold reward whenever ghost arbitrage is available. theoretical
    Source sentence
    a ghost-arbitrage incentive theorem showing that naive PPO-type RL is structurally driven to increase E[r⊥] whenever ghost arbitrage is available
  • Law‑strength trade‑off theorem proves that increasing the weight on law penalties inevitably worsens P&L beyond a quantifiable threshold. theoretical
    Source sentence
    a law-strength trade-off theorem establishing that increasing the weight on law penalties inevitably worsens P&L beyond a quantifiable threshold
  • No‑free‑lunch theorem states that under a law‑consistent world model and law‑aligned strategy class, unconstrained law‑seeking RL cannot Pareto‑dominate structural baselines on P&L, penalties, and GFI. theoretical
    Source sentence
    a no-free-lunch theorem: under a law-consistent world model and law-aligned strategy class, unconstrained law-seeking RL cannot Pareto-dominate structural baselines on P&L, penalties, and GFI
  • Empirically, all law‑seeking RL variants underperform structural baselines, occupying high‑penalty, high‑GFI regions, while baselines lie on the empirical law‑strength frontier. performance
    Source sentence
    all law‑seeking RL variants underperform and move into high-penalty, high-GFI regions. Volatility thus provides a concrete case where reward shaping with verifiable penalties is insufficient for robust law alignment.
  • In the shock regime, RL policies in the same law‑penalty band have negative Sharpe and GFI > 1.5, whereas Zero‑Hedge maintains high Sharpe and GFI ≈0. performance
    Source sentence
    Within the law-penalty band [0.0053, 0.0057] highlighted, Zero-Hedge lies near the lower boundary with high Sharpe and GFI ≈0, while the closest RL variants in the band have Sharpe < 0 and GFI > 1.5

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 28 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (80)

  1. Pricing with a Smile (1994)
  2. The Volatility Surface: A Practitioner’s Guide (2006)
  3. Semiparametric Modeling of Implied Volatility (2005)
  4. Option Valuation Using the Fast Fourier Transform (1999)
  5. The Moment Formula for Implied Volatility at Extreme Strikes (2004)
  6. Financial Modelling with Jump Processes (2004)
  7. Stochastic Volatility Modeling (2016)
  8. Arbitrage-Free SVI Volatility Surfaces (2014)
and 72 more
  1. Pricing under Rough Volatility (2016)
  2. Deep Learning Volatility: Deep Calibration of Rough Stochastic Volatility Models (2021)
  3. Deep Learning Volatility (2021)
  4. Calibration of Local Stochastic Volatility Models to Market Data (2015)
  5. Linking Vanillas and VIX Options: A Constrained Optimization Approach (2015)
  6. A Sparse Modeling Approach to the Arbitrage-Free Interpolation of Plain-Vanilla Option Prices and Implied Volatilities (2023)
  7. Neural Network-Based Option Pricing (2020) — link
  8. A Note on Sufficient Conditions for No Arbitrage (2005)
  9. Interpolation Methods for Curve Construction (2006)
  10. Term-Structure Models: A Graduate Course (2009)
  11. Consistency Problems for Heath–Jarrow–Morton Interest Rate Models (2001)
  12. Interest Rate Dynamics and Consistent Forward Rate Curves (1999)
  13. Coherent Measures of Risk (1999)
  14. Optimization of Conditional Value-at-Risk (2000)
  15. Algorithmic and High-Frequency Trading (2015)
  16. A Century of Evidence on Trend-Following Investing (2017)
  17. Deep Reinforcement Learning for Trading (2020)
  18. Convex Analysis and Monotone Operator Theory in Hilbert Spaces (2011)
  19. Stochastic Calculus and Applications (2015)
  20. Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples (2006)
  21. Testing the Manifold Hypothesis (2016)
  22. Solving High-Dimensional Partial Differential Equations Using Deep Learning (2018)
  23. Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-Order Backward SDE s (2019)
  24. Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019)
  25. Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019)
  26. Physics-Informed Machine Learning (2021)
  27. Machine Learning and the Physical Sciences (2019)
  28. Integrating Physics-Based Modeling with Machine Learning: A Survey (2020)
  29. Solving Stochastic Differential Equations and Kolmogorov Equations by Deep Learning (2021)
  30. Message Passing Neural PDE Solvers (2022)
  31. Relational Inductive Biases, Deep Learning, and Graph Networks (2018) — link
  32. World Models (2018) — link
  33. Learning Latent Dynamics for Planning from Pixels (2019)
  34. Dream to Control: Learning Behaviors by Latent Imagination (2020)
  35. Mastering Diverse Domains with World Models (2023) — link
  36. Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models (2018)
  37. When to Trust Your Model: Model-Based Policy Optimization (2019)
  38. Model-Based Reinforcement Learning for Atari (2020)
  39. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (2020)
  40. Human-Level Control Through Deep Reinforcement Learning (2015)
  41. Reinforcement Learning: An Introduction (2018)
  42. Actor-Critic Algorithms (2000)
  43. Bias in Natural Actor-Critic Algorithms (2014)
  44. High-Dimensional Continuous Control Using Generalized Advantage Estimation (2016)
  45. Proximal Policy Optimization Algorithms (2017)
  46. Constrained Markov Decision Processes (1999)
  47. Constrained Policy Optimization (2017)
  48. Mean-Variance Optimization in Markov Decision Processes (2011)
  49. Risk-Sensitive and Robust Decision-Making: A CVaR Optimization Approach (2015)
  50. A Survey of Multi-Objective Sequential Decision-Making (2013)
  51. A Unified View of Entropy-Regularized Markov Decision Processes (2017)
  52. A Theory of Regularized Markov Decision Processes (2019)
  53. Towards Safe Reinforcement Learning: A Survey and Outlook (2021) — link
  54. Regularized Policy Optimization in MDP s with Constraints (2021)
  55. Learning to Trade via Direct Reinforcement (2001)
  56. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2016)
  57. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
  58. FinRL : A Deep Reinforcement Learning Library for Quantitative Finance (2021)
  59. Dynamic Replication and Hedging: A Reinforcement Learning Approach (2019)
  60. Deep Hedging (2019)
  61. Deep Hedging (2019)
  62. Deep Hedging (2019)
  63. Deep Hedging of Derivatives with Transaction Costs and Different Risk Criteria (2020)
  64. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks (2020)
  65. A Transformer-Based Framework for Multivariate Time Series Representation Learning (2021)
  66. Categorizing Variants of Goodhart’s Law (2019) — link
  67. Concrete Problems in AI Safety (2016) — link
  68. AI Safety Gridworlds (2017)
  69. Reinforcement Learning with a Corrupted Reward Channel (2017) — link
  70. Hidden Incentives in Deep Reinforcement Learning (2020)
  71. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022) — link
  72. A Practical Guide to Multi-Objective Reinforcement Learning and Planning (2022)

0 of 80 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.