Paper: arXiv 2510.09247

Authors: Zofia Bracha, Paweł Sakowski, Jakub Michańków

Abstract

This paper explores the application of deep Q-learning to hedging at-the-money options on the S&P500 index. We develop an agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, trained to simulate hedging decisions without making explicit model assumptions on price dynamics. The agent was trained on historical intraday prices of S&P500 call options across years 2004–2024, using a single time series of six predictor variables: option price, underlying asset price, moneyness, time to maturity, realized volatility, and current hedge position. A walk-forward procedure was applied for training, which led to nearly 17~years of out-of-sample evaluation. The performance of the deep reinforcement learning (DRL) agent is benchmarked against the Black–Scholes delta-hedging strategy over the same period. We assess both approaches using metrics such as annualized return, volatility, information ratio, and Sharpe ratio. To test the models’ adaptability, we performed simulations across varying market conditions and added constraints such as transaction costs and risk-awareness penalties. Our results show that the DRL agent can outperform traditional hedging methods, particularly in volatile or high-cost environments, highlighting its robustness and flexibility in practical trading contexts. While the agent consistently outperforms delta-hedging, its performance deteriorates when the risk-awareness parameter is higher. We also observed that the longer the time interval used for volatility estimation, the more stable the results.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced deep reinforcement learning algorithms like TD3 and discusses underlying theory (Bellman equations, Q-learning), contributing to high math complexity. Empirical rigor is strong due to extensive historical intraday data (2004-2024), walk-forward out-of-sample testing, multiple performance metrics, and sensitivity analyses on transaction costs and volatility windows.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Apply Deep RL to<br>ATM S&P 500 Options Hedging"] --> B["Data & Methodology"]
  
  subgraph B ["Data & Methodology"]
      B1["Historical Data<br>2004-2024<br>6 Predictors"]
      B2["Walk-Forward<br>Training Procedure"]
      B3["Algorithms: TD3 (DRL)<br>vs. Black-Scholes Delta"]
  end

  B --> C["Computational Process"]
  
  subgraph C ["Computational Process"]
      C1["Agent Training<br>Simulate Hedging Decisions"]
      C2["Constraints & Scenarios<br>Transaction Costs<br>Risk Penalties"]
  end

  C --> D["Key Findings & Outcomes"]
  
  subgraph D ["Key Findings & Outcomes"]
      D1["DRL Outperforms Delta<br>Specifically in Volatile/<br>High-Cost Environments"]
      D2["Performance Deteriorates<br>with High Risk-Awareness"]
      D3["Longer Volatility Estimation<br>Intervals = More Stable Results"]
  end

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a deep reinforcement learning agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for hedging at‑the‑money S&P 500 options.

Source sentence
We develop an agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, trained to simulate hedging decisions without making explicit model assumptions on price dynamics.

Assumes 5

  • A constant risk‑free rate of 2% is used for all periods. market
    Source sentence
    Across all periods we assumed 2% risk free rate.
  • Rebalancing occurs every 30 minutes. execution
    Source sentence
    current setup of trading every 30 minutes drastically increase the costs.
  • Realized volatility is estimated using a rolling standard deviation of log‑returns. data
    Source sentence
    In this paper we employ a realized volatility framework . First, asset prices
  • A walk‑forward training procedure is used to generate nearly 17 years of out‑of‑sample evaluation. other
    Source sentence
    A walk‑forward procedure was applied for training, which lead to nearly 17 years of out‑of‑sample evaluation.
Show 1 more
  • The benchmark delta‑hedging strategy assumes continuous trading with no transaction costs or liquidity constraints (Black‑Scholes model). model
    Source sentence
    It assumes a market with constant volatility and interest rates, allowance of continuous trading without transaction costs or liquidity constraints.

Data

Backtest sample: 2004 – 2024 · S&P 500 options · options · intraday

  • Historical intraday prices of S&P 500 call options options unknown
    2004‑2024 · at‑the‑money S&P 500 call options
    Source sentence
    The agent was trained on historical intraday prices of S&P 500 call options across years 2004 to 2024, using a single time series of six predictor variables: option price, underlying asset price, moneyness, time to maturity, realized volatility, and current hedge position.

Recognised sources: Cboe

Costs

Modeled Costs are part of the reported headline results.

Costs are included via a flat parameter c (e.g., c = 0.1% in the base case) in the PnL calculation.

Source sentence
c represents transaction costs, HS holding of stock, S stock price, HO holding of option, and finally C represents option price.

Finds 4

Reported performance (TD3 deep reinforcement learning hedging agent): Sharpe -0.42 (benchmark -0.42) · annual return -0.4% · total return -45.5% · max drawdown 75.3%

  • The DRL agent outperforms the Black‑Scholes delta‑hedging benchmark across all reported metrics in the base case. performance
    Source sentence
    The results in the Table 2 show superior performance of the DRL agent over the classical delta hedging benchmark model. The DRL managed to get positive Information ratio, despite heavy drawdowns and negative returns.
  • When transaction costs are reduced, the DRL agent’s performance improves and can become positive, while the benchmark may still lag. performance
    Source sentence
    When the costs are low, the benchmark performs better both in terms of returns or information ratio, but also experiences lower drawdowns.
  • Higher risk‑awareness (risk‑penalty parameter) reduces the DRL agent’s returns and performance. performance
    Source sentence
    During the sensitivity analysis, we observed that the increase in risk aversion reduces returns, while shorter volatility estimation intervals make the model more responsive to jumps
  • The DRL framework shows particular advantage in volatile or high‑cost environments, adapting to market conditions where delta hedging deteriorates sharply. performance
    Source sentence
    Our results show that the DRL agent can outperform traditional hedging methods, particularly in volatile or high‑cost environments, highlighting its robustness and flexibility in practical trading contexts.

Checked by others

Supported by

  • Empirical Analysis of the Model-Free Valuation Approach: Hedging Gaps, Conservatism, and Trading Opportunities (2025)
    On the question: How does increasing the risk‑parameter (c or risk‑penalty) affect average profit and risk‑adjusted returns?
    Compare the findings

    This paper: Higher risk‑awareness (risk‑penalty parameter) reduces the DRL agent’s returns and performance.

    That paper: Increasing the risk parameter c reduces average profit and risk‑adjusted returns.

    Both findings report that higher risk‑parameter reduces returns: A shows c lowers average profit and risk‑adjusted returns; B shows risk‑penalty lowers DRL agent returns. Markets differ (model‑free valuation vs DRL on AT

  • A Game of Competition for Risk (2023)
    On the question: How does increasing a risk‑penalty affect average risk taken and resulting returns?
    Compare the findings

    This paper: Higher risk‑awareness (risk‑penalty parameter) reduces the DRL agent’s returns and performance.

    That paper: Penalties reduce both the average risk taken by players and their total rewards.

    Both findings show that higher risk‑penalties lower the average risk and also reduce total rewards or performance.

Code

https://github.com/zof-br/DRL_hedging — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (64)

  1. The Pricing of Options and Corporate Liabilities (1973)
  2. Neural Networks for Delta Hedging (2021) — link
  3. On Quadratic Hedging in Continuous Time (2000) — link
  4. Variance-Optimal Hedging in Discrete Time (1995) — link
  5. Deep hedging of derivatives using reinforcement learning (2021) — link
  6. Empirical deep hedging (2023)
  7. Deep hedging (2019)
  8. The relation between implied and realized volatility (1998)
and 56 more
  1. Portfolio Selection (1952)
  2. Portfolio Selection: Efficient Diversification of Investments (1959)
  3. Predicting implied volatility: Cross-sectional efficiency and information content (2009)
  4. The Predictive Power of Implied and Realized Volatility on the Swedish Market (2023) — link
  5. PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019)
  6. Deep bellman hedging (2022) — link
  7. Deep Hedging: Learning to Remove the Drift under Trading Frictions with Minimal Equivalent Near-Martingale Measures (2021) — link
  8. Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversions (2022) — link
  9. Is the difference between deep hedging and delta hedging a statistical arbitrage? — → 2407.14736
  10. Addressing Function Approximation Error in Actor-Critic Methods (2018) — link
  11. Deep Reinforcement Learning Algorithms for Option Hedging — → 2504.05521
  12. A Closed-Form Solution for Options with Stochastic Volatility with Applications to Bond and Currency Options (1993) — link
  13. Mutual Fund Performance (1966) — link
  14. Dynamic replication and hedging: A reinforcement learning approach (2020)
  15. Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation (1982) — link
  16. Generalized Autoregressive Conditional Heteroskedasticity (1986) — link
  17. Managing Smile Risk (2002)
  18. Deep Reinforcement Learning with Double Q-learning (2015) — link
  19. On the Theory of Dynamic Programming (1952) — link
  20. Deep Hedging with Options Using the Implied Volatility Surface — → 2504.06208
  21. Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback Information — → 2407.21138
  22. Machine Learning in Finance: From Theory to Practice (2022)
  23. Convergence analysis of machine learning algorithms for option pricing and hedging (2022)
  24. Hedging with Linear Regressions and Neural Networks (2021)
  25. Neural networks for option pricing and hedging: a literature review (2020) — link
  26. Reinforcement learning for trading and hedging in continuous time (2023)
  27. QLBS: Q-Learner in the Black–Scholes(-Merton) worlds (2020)
  28. Theory of Rational Option Pricing (1973)
  29. Options, Futures, and Other Derivatives (2018)
  30. Paul Wilmott on Quantitative Finance (2006)
  31. Reinforcement Learning: An Introduction (2018)
  32. Temporal Difference Learning and TD-Gammon (1995)
  33. Human-level control through deep reinforcement learning (2015)
  34. Actor-critic algorithms (2000)
  35. Hedging of non-redundant contingent claims (1986)
  36. Hedging of contingent claims under incomplete information (1991)
  37. Quantile hedging (1999)
  38. Hedging derivative securities and incomplete markets: an $\varepsilon$-arbitrage approach (2001)
  39. Robust risk-aware option hedging — → 2303.15216
  40. Learning to hedge without price process modeling (2023)
  41. Optimization of conditional value-at-risk (2000)
  42. A nonparametric approach to pricing and hedging derivative securities via learning networks (1994)
  43. Machine learning approaches to hedging (2022)
  44. Deterministic Policy Gradient Algorithms (2014)
  45. On the theory of Brownian Motion (1930)
  46. Modeling and Forecasting Realized Volatility (2003) — link
  47. Continuous Control with Deep Reinforcement Learning (2016) — link
  48. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992)
  49. Neuro-Dynamic Programming (1996)
  50. Dynamic Programming and Optimal Control (2012)
  51. Markov Decision Processes: Discrete Stochastic Dynamic Programming (2005)
  52. Proximal Policy Optimization Algorithms (2017) — link
  53. Learning to Predict by the Methods of Temporal Differences (1988) — link
  54. Q-learning (1992) — link
  55. Learning From Delayed Rewards (1989)
  56. A Comparison of Long Short-Term Memory and Gated Recurrent Unit Models’ Architectures with Novel Walk-Forward Approach to Algorithmic Investment Strategy (2023) — link

5 of 64 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.