Paper: arXiv 2510.09247
Authors: Zofia Bracha, Paweł Sakowski, Jakub Michańków
Abstract
This paper explores the application of deep Q-learning to hedging at-the-money options on the S&P500 index. We develop an agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, trained to simulate hedging decisions without making explicit model assumptions on price dynamics. The agent was trained on historical intraday prices of S&P500 call options across years 2004–2024, using a single time series of six predictor variables: option price, underlying asset price, moneyness, time to maturity, realized volatility, and current hedge position. A walk-forward procedure was applied for training, which led to nearly 17~years of out-of-sample evaluation. The performance of the deep reinforcement learning (DRL) agent is benchmarked against the Black–Scholes delta-hedging strategy over the same period. We assess both approaches using metrics such as annualized return, volatility, information ratio, and Sharpe ratio. To test the models’ adaptability, we performed simulations across varying market conditions and added constraints such as transaction costs and risk-awareness penalties. Our results show that the DRL agent can outperform traditional hedging methods, particularly in volatile or high-cost environments, highlighting its robustness and flexibility in practical trading contexts. While the agent consistently outperforms delta-hedging, its performance deteriorates when the risk-awareness parameter is higher. We also observed that the longer the time interval used for volatility estimation, the more stable the results.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 7.5/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs advanced deep reinforcement learning algorithms like TD3 and discusses underlying theory (Bellman equations, Q-learning), contributing to high math complexity. Empirical rigor is strong due to extensive historical intraday data (2004-2024), walk-forward out-of-sample testing, multiple performance metrics, and sensitivity analyses on transaction costs and volatility windows.
Research Flowchart
flowchart TD
A["Research Goal<br>Apply Deep RL to<br>ATM S&P 500 Options Hedging"] --> B["Data & Methodology"]
subgraph B ["Data & Methodology"]
B1["Historical Data<br>2004-2024<br>6 Predictors"]
B2["Walk-Forward<br>Training Procedure"]
B3["Algorithms: TD3 (DRL)<br>vs. Black-Scholes Delta"]
end
B --> C["Computational Process"]
subgraph C ["Computational Process"]
C1["Agent Training<br>Simulate Hedging Decisions"]
C2["Constraints & Scenarios<br>Transaction Costs<br>Risk Penalties"]
end
C --> D["Key Findings & Outcomes"]
subgraph D ["Key Findings & Outcomes"]
D1["DRL Outperforms Delta<br>Specifically in Volatile/<br>High-Cost Environments"]
D2["Performance Deteriorates<br>with High Risk-Awareness"]
D3["Longer Volatility Estimation<br>Intervals = More Stable Results"]
end
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a deep reinforcement learning agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for hedging at‑the‑money S&P 500 options.
Source sentence
We develop an agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, trained to simulate hedging decisions without making explicit model assumptions on price dynamics.
Assumes 5
- A constant risk‑free rate of 2% is used for all periods. market
Source sentence
Across all periods we assumed 2% risk free rate.
- Rebalancing occurs every 30 minutes. execution
Source sentence
current setup of trading every 30 minutes drastically increase the costs.
- Realized volatility is estimated using a rolling standard deviation of log‑returns. data
Source sentence
In this paper we employ a realized volatility framework . First, asset prices
- A walk‑forward training procedure is used to generate nearly 17 years of out‑of‑sample evaluation. other
Source sentence
A walk‑forward procedure was applied for training, which lead to nearly 17 years of out‑of‑sample evaluation.
Show 1 more
- The benchmark delta‑hedging strategy assumes continuous trading with no transaction costs or liquidity constraints (Black‑Scholes model). model
Source sentence
It assumes a market with constant volatility and interest rates, allowance of continuous trading without transaction costs or liquidity constraints.
Data
2004 – 2024 · S&P 500 options · options · intraday
- Historical intraday prices of S&P 500 call options options unknown
Source sentence
The agent was trained on historical intraday prices of S&P 500 call options across years 2004 to 2024, using a single time series of six predictor variables: option price, underlying asset price, moneyness, time to maturity, realized volatility, and current hedge position.
Cboe
Costs
Modeled
Costs are included via a flat parameter c (e.g., c = 0.1% in the base case) in the PnL calculation.
Source sentence
c represents transaction costs, HS holding of stock, S stock price, HO holding of option, and finally C represents option price.
Finds 4
Sharpe -0.42 (benchmark -0.42) · annual return -0.4% · total return -45.5% · max drawdown 75.3%
- The DRL agent outperforms the Black‑Scholes delta‑hedging benchmark across all reported metrics in the base case. performance
Source sentence
The results in the Table 2 show superior performance of the DRL agent over the classical delta hedging benchmark model. The DRL managed to get positive Information ratio, despite heavy drawdowns and negative returns.
- When transaction costs are reduced, the DRL agent’s performance improves and can become positive, while the benchmark may still lag. performance
Source sentence
When the costs are low, the benchmark performs better both in terms of returns or information ratio, but also experiences lower drawdowns.
- Higher risk‑awareness (risk‑penalty parameter) reduces the DRL agent’s returns and performance. performance
Source sentence
During the sensitivity analysis, we observed that the increase in risk aversion reduces returns, while shorter volatility estimation intervals make the model more responsive to jumps
- The DRL framework shows particular advantage in volatile or high‑cost environments, adapting to market conditions where delta hedging deteriorates sharply. performance
Source sentence
Our results show that the DRL agent can outperform traditional hedging methods, particularly in volatile or high‑cost environments, highlighting its robustness and flexibility in practical trading contexts.
Checked by others
Supported by
- Empirical Analysis of the Model-Free Valuation Approach: Hedging Gaps, Conservatism, and Trading Opportunities
Compare the findings
This paper: Higher risk‑awareness (risk‑penalty parameter) reduces the DRL agent’s returns and performance.
That paper: Increasing the risk parameter c reduces average profit and risk‑adjusted returns.
- A Game of Competition for Risk
Compare the findings
This paper: Higher risk‑awareness (risk‑penalty parameter) reduces the DRL agent’s returns and performance.
That paper: Penalties reduce both the average risk taken by players and their total rewards.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (64)
- The Pricing of Options and Corporate Liabilities (1973)
- Neural Networks for Delta Hedging (2021) — link
- On Quadratic Hedging in Continuous Time (2000) — link
- Variance-Optimal Hedging in Discrete Time (1995) — link
- Deep hedging of derivatives using reinforcement learning (2021) — link
- Empirical deep hedging (2023)
- Deep hedging (2019)
- The relation between implied and realized volatility (1998)
and 56 more
- Portfolio Selection (1952)
- Portfolio Selection: Efficient Diversification of Investments (1959)
- Predicting implied volatility: Cross-sectional efficiency and information content (2009)
- The Predictive Power of Implied and Realized Volatility on the Swedish Market (2023) — link
- PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019)
- Deep bellman hedging (2022) — link
- Deep Hedging: Learning to Remove the Drift under Trading Frictions with Minimal Equivalent Near-Martingale Measures (2021) — link
- Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversions (2022) — link
- Is the difference between deep hedging and delta hedging a statistical arbitrage? — → 2407.14736
- Addressing Function Approximation Error in Actor-Critic Methods (2018) — link
- Deep Reinforcement Learning Algorithms for Option Hedging — → 2504.05521
- A Closed-Form Solution for Options with Stochastic Volatility with Applications to Bond and Currency Options (1993) — link
- Mutual Fund Performance (1966) — link
- Dynamic replication and hedging: A reinforcement learning approach (2020)
- Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation (1982) — link
- Generalized Autoregressive Conditional Heteroskedasticity (1986) — link
- Managing Smile Risk (2002)
- Deep Reinforcement Learning with Double Q-learning (2015) — link
- On the Theory of Dynamic Programming (1952) — link
- Deep Hedging with Options Using the Implied Volatility Surface — → 2504.06208
- Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback Information — → 2407.21138
- Machine Learning in Finance: From Theory to Practice (2022)
- Convergence analysis of machine learning algorithms for option pricing and hedging (2022)
- Hedging with Linear Regressions and Neural Networks (2021)
- Neural networks for option pricing and hedging: a literature review (2020) — link
- Reinforcement learning for trading and hedging in continuous time (2023)
- QLBS: Q-Learner in the Black–Scholes(-Merton) worlds (2020)
- Theory of Rational Option Pricing (1973)
- Options, Futures, and Other Derivatives (2018)
- Paul Wilmott on Quantitative Finance (2006)
- Reinforcement Learning: An Introduction (2018)
- Temporal Difference Learning and TD-Gammon (1995)
- Human-level control through deep reinforcement learning (2015)
- Actor-critic algorithms (2000)
- Hedging of non-redundant contingent claims (1986)
- Hedging of contingent claims under incomplete information (1991)
- Quantile hedging (1999)
- Hedging derivative securities and incomplete markets: an $\varepsilon$-arbitrage approach (2001)
- Robust risk-aware option hedging — → 2303.15216
- Learning to hedge without price process modeling (2023)
- Optimization of conditional value-at-risk (2000)
- A nonparametric approach to pricing and hedging derivative securities via learning networks (1994)
- Machine learning approaches to hedging (2022)
- Deterministic Policy Gradient Algorithms (2014)
- On the theory of Brownian Motion (1930)
- Modeling and Forecasting Realized Volatility (2003) — link
- Continuous Control with Deep Reinforcement Learning (2016) — link
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992)
- Neuro-Dynamic Programming (1996)
- Dynamic Programming and Optimal Control (2012)
- Markov Decision Processes: Discrete Stochastic Dynamic Programming (2005)
- Proximal Policy Optimization Algorithms (2017) — link
- Learning to Predict by the Methods of Temporal Differences (1988) — link
- Q-learning (1992) — link
- Learning From Delayed Rewards (1989)
- A Comparison of Long Short-Term Memory and Gated Recurrent Unit Models’ Architectures with Novel Walk-Forward Approach to Algorithmic Investment Strategy (2023) — link
5 of 64 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.