Paper: arXiv 2511.17304
Authors: Jian’an Zhang
Abstract
We study reinforcement learning (RL) on volatility surfaces through the lens of Scientific AI. We ask whether axiomatic no-arbitrage laws, imposed as soft penalties on a learned world model, can reliably align high-capacity RL agents, or mainly create Goodhart-style incentives to exploit model errors. From classical static no-arbitrage conditions we build a finite-dimensional convex volatility law manifold of admissible total-variance surfaces, together with a metric law-penalty functional and a Graceful Failure Index (GFI) that normalizes law degradation under shocks. A synthetic generator produces law-consistent trajectories, while a recurrent neural world model trained without law regularization exhibits structured off-manifold errors. On this testbed we define a Goodhart decomposition (r = r^{"\mathcal{M"}} + r^\perp), where (r^\perp) is ghost arbitrage from off-manifold prediction error. We prove a ghost-arbitrage incentive theorem for PPO-type agents, a law-strength trade-off theorem showing that stronger penalties eventually worsen P&L, and a no-free-lunch theorem: under a law-consistent world model and law-aligned strategy class, unconstrained law-seeking RL cannot Pareto-dominate structural baselines on P&L, penalties, and GFI. In experiments on an SPX/VIX-like world model, simple structural strategies form the empirical law-strength frontier, while all law-seeking RL variants underperform and move into high-penalty, high-GFI regions. Volatility thus provides a concrete case where reward shaping with verifiable penalties is insufficient for robust law alignment.
Complexity vs Empirical Score
- Math Complexity: 9.5/10
- Empirical Rigor: 4.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper is dense with advanced theoretical results, including proofs of incentive and trade-off theorems, convex manifolds, and Goodhart decomposition, indicating very high mathematical complexity. Empirically, while it uses a synthetic testbed and specific metrics (e.g., GFI), the work is primarily theoretical and diagnostic with no real-world data, backtests, or code, placing it lower on empirical rigor.
Research Flowchart
flowchart TD
A["Research Goal"] --> B["Data: Law-Consistent Trajectories"]
B --> C["Methodology: RL on World Model"]
C --> D["Computations"]
subgraph D
D1["Train RNN World Model"]
D2["Apply Law Penalty"]
D3["Run PPO Agent"]
end
D --> E{"Outcomes"}
E --> F["Ghost Arbitrage Incentive"]
E --> G["Law-Strength Trade-off"]
E --> H["No Free Lunch on Volatility Frontier"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a finite‑dimensional convex volatility law manifold with a metric‑based law‑penalty functional and a Graceful Failure Index, and introduces a Goodhart decomposition of reward into on‑manifold and ghost arbitrage components.
Source sentence
we construct a finite-dimensional convex volatility law manifold of admissible total-variance surfaces, together with a metric-based law-penalty functional and a domain-agnostic Graceful Failure Index (GFI) that normalizes law degradation under shocks.
Assumes 5
- The structural class S is non‑empty, convex, and law‑consistent with bounded law penalties and GFI. model
Source sentence
Assumption 3 (Structural class and world-model properties). We assume: 1. Law-consistent structural class. The structural class S ⊂Π is non-empty, convex, and law-consistent in the sense that for all s ∈S, E[Lϕ(s)] ≤LSmax and GFI(s) ≤GFISmax
- The unconstrained policy class Π is rich enough to strictly contain S and to reach off‑manifold regions. model
Source sentence
Rich unconstrained class. The unconstrained policy class Π is rich enough to strictly contain S and to reach off-manifold regions
- The synthetic generator G⋆ produces trajectories that are exactly law‑consistent (each wt lies in Mvol). data
Source sentence
We call G⋆ a law-consistent generator if the trajectory (wt) it induces satisfies this condition for M = Mvol.
- The recurrent neural world model is trained without any law‑penalty regularization, using only predictive MSE loss. model
Source sentence
Crucially, no law penalties are used when training the world model: the loss is purely predictive.
Show 1 more
- Shock regime changes the distribution of trajectories (via long‑variance and spot‑variance multipliers) while keeping the underlying axioms unchanged. market
Source sentence
To probe robustness and graceful failure, we introduce a shock regime in which the same axioms hold, but the volatility regime is stressed.
Data
- Law‑consistent synthetic generator (G⋆) synthetic unknown
Source sentence
A synthetic generator produces trajectories that are exactly law-consistent, while a recurrent neural world model is trained without law regularization and therefore predicts surfaces that deviate from the law manifold in structured ways.
Costs
Not modeled
Finds 5
- Ghost‑arbitrage incentive theorem shows naive PPO‑type RL is driven to increase off‑manifold reward whenever ghost arbitrage is available. theoretical
Source sentence
a ghost-arbitrage incentive theorem showing that naive PPO-type RL is structurally driven to increase E[r⊥] whenever ghost arbitrage is available
- Law‑strength trade‑off theorem proves that increasing the weight on law penalties inevitably worsens P&L beyond a quantifiable threshold. theoretical
Source sentence
a law-strength trade-off theorem establishing that increasing the weight on law penalties inevitably worsens P&L beyond a quantifiable threshold
- No‑free‑lunch theorem states that under a law‑consistent world model and law‑aligned strategy class, unconstrained law‑seeking RL cannot Pareto‑dominate structural baselines on P&L, penalties, and GFI. theoretical
Source sentence
a no-free-lunch theorem: under a law-consistent world model and law-aligned strategy class, unconstrained law-seeking RL cannot Pareto-dominate structural baselines on P&L, penalties, and GFI
- Empirically, all law‑seeking RL variants underperform structural baselines, occupying high‑penalty, high‑GFI regions, while baselines lie on the empirical law‑strength frontier. performance
Source sentence
all law‑seeking RL variants underperform and move into high-penalty, high-GFI regions. Volatility thus provides a concrete case where reward shaping with verifiable penalties is insufficient for robust law alignment.
- In the shock regime, RL policies in the same law‑penalty band have negative Sharpe and GFI > 1.5, whereas Zero‑Hedge maintains high Sharpe and GFI ≈0. performance
Source sentence
Within the law-penalty band [0.0053, 0.0057] highlighted, Zero-Hedge lies near the lower boundary with high Sharpe and GFI ≈0, while the closest RL variants in the band have Sharpe < 0 and GFI > 1.5
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 28 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (80)
- Pricing with a Smile (1994)
- The Volatility Surface: A Practitioner’s Guide (2006)
- Semiparametric Modeling of Implied Volatility (2005)
- Option Valuation Using the Fast Fourier Transform (1999)
- The Moment Formula for Implied Volatility at Extreme Strikes (2004)
- Financial Modelling with Jump Processes (2004)
- Stochastic Volatility Modeling (2016)
- Arbitrage-Free SVI Volatility Surfaces (2014)
and 72 more
- Pricing under Rough Volatility (2016)
- Deep Learning Volatility: Deep Calibration of Rough Stochastic Volatility Models (2021)
- Deep Learning Volatility (2021)
- Calibration of Local Stochastic Volatility Models to Market Data (2015)
- Linking Vanillas and VIX Options: A Constrained Optimization Approach (2015)
- A Sparse Modeling Approach to the Arbitrage-Free Interpolation of Plain-Vanilla Option Prices and Implied Volatilities (2023)
- Neural Network-Based Option Pricing (2020) — link
- A Note on Sufficient Conditions for No Arbitrage (2005)
- Interpolation Methods for Curve Construction (2006)
- Term-Structure Models: A Graduate Course (2009)
- Consistency Problems for Heath–Jarrow–Morton Interest Rate Models (2001)
- Interest Rate Dynamics and Consistent Forward Rate Curves (1999)
- Coherent Measures of Risk (1999)
- Optimization of Conditional Value-at-Risk (2000)
- Algorithmic and High-Frequency Trading (2015)
- A Century of Evidence on Trend-Following Investing (2017)
- Deep Reinforcement Learning for Trading (2020)
- Convex Analysis and Monotone Operator Theory in Hilbert Spaces (2011)
- Stochastic Calculus and Applications (2015)
- Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples (2006)
- Testing the Manifold Hypothesis (2016)
- Solving High-Dimensional Partial Differential Equations Using Deep Learning (2018)
- Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-Order Backward SDE s (2019)
- Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019)
- Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019)
- Physics-Informed Machine Learning (2021)
- Machine Learning and the Physical Sciences (2019)
- Integrating Physics-Based Modeling with Machine Learning: A Survey (2020)
- Solving Stochastic Differential Equations and Kolmogorov Equations by Deep Learning (2021)
- Message Passing Neural PDE Solvers (2022)
- Relational Inductive Biases, Deep Learning, and Graph Networks (2018) — link
- World Models (2018) — link
- Learning Latent Dynamics for Planning from Pixels (2019)
- Dream to Control: Learning Behaviors by Latent Imagination (2020)
- Mastering Diverse Domains with World Models (2023) — link
- Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models (2018)
- When to Trust Your Model: Model-Based Policy Optimization (2019)
- Model-Based Reinforcement Learning for Atari (2020)
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (2020)
- Human-Level Control Through Deep Reinforcement Learning (2015)
- Reinforcement Learning: An Introduction (2018)
- Actor-Critic Algorithms (2000)
- Bias in Natural Actor-Critic Algorithms (2014)
- High-Dimensional Continuous Control Using Generalized Advantage Estimation (2016)
- Proximal Policy Optimization Algorithms (2017)
- Constrained Markov Decision Processes (1999)
- Constrained Policy Optimization (2017)
- Mean-Variance Optimization in Markov Decision Processes (2011)
- Risk-Sensitive and Robust Decision-Making: A CVaR Optimization Approach (2015)
- A Survey of Multi-Objective Sequential Decision-Making (2013)
- A Unified View of Entropy-Regularized Markov Decision Processes (2017)
- A Theory of Regularized Markov Decision Processes (2019)
- Towards Safe Reinforcement Learning: A Survey and Outlook (2021) — link
- Regularized Policy Optimization in MDP s with Constraints (2021)
- Learning to Trade via Direct Reinforcement (2001)
- Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2016)
- A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
- FinRL : A Deep Reinforcement Learning Library for Quantitative Finance (2021)
- Dynamic Replication and Hedging: A Reinforcement Learning Approach (2019)
- Deep Hedging (2019)
- Deep Hedging (2019)
- Deep Hedging (2019)
- Deep Hedging of Derivatives with Transaction Costs and Different Risk Criteria (2020)
- DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks (2020)
- A Transformer-Based Framework for Multivariate Time Series Representation Learning (2021)
- Categorizing Variants of Goodhart’s Law (2019) — link
- Concrete Problems in AI Safety (2016) — link
- AI Safety Gridworlds (2017)
- Reinforcement Learning with a Corrupted Reward Channel (2017) — link
- Hidden Incentives in Deep Reinforcement Learning (2020)
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022) — link
- A Practical Guide to Multi-Objective Reinforcement Learning and Planning (2022)
0 of 80 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.