Paper: arXiv 2510.09247

Authors: Zofia Bracha, Paweł Sakowski, Jakub Michańków

Abstract

This paper explores the application of deep Q-learning to hedging at-the-money options on the S&P500 index. We develop an agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, trained to simulate hedging decisions without making explicit model assumptions on price dynamics. The agent was trained on historical intraday prices of S&P500 call options across years 2004–2024, using a single time series of six predictor variables: option price, underlying asset price, moneyness, time to maturity, realized volatility, and current hedge position. A walk-forward procedure was applied for training, which led to nearly 17~years of out-of-sample evaluation. The performance of the deep reinforcement learning (DRL) agent is benchmarked against the Black–Scholes delta-hedging strategy over the same period. We assess both approaches using metrics such as annualized return, volatility, information ratio, and Sharpe ratio. To test the models’ adaptability, we performed simulations across varying market conditions and added constraints such as transaction costs and risk-awareness penalties. Our results show that the DRL agent can outperform traditional hedging methods, particularly in volatile or high-cost environments, highlighting its robustness and flexibility in practical trading contexts. While the agent consistently outperforms delta-hedging, its performance deteriorates when the risk-awareness parameter is higher. We also observed that the longer the time interval used for volatility estimation, the more stable the results.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced deep reinforcement learning algorithms like TD3 and discusses underlying theory (Bellman equations, Q-learning), contributing to high math complexity. Empirical rigor is strong due to extensive historical intraday data (2004-2024), walk-forward out-of-sample testing, multiple performance metrics, and sensitivity analyses on transaction costs and volatility windows.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Apply Deep RL to<br>ATM S&P 500 Options Hedging"] --> B["Data & Methodology"]
  
  subgraph B ["Data & Methodology"]
      B1["Historical Data<br>2004-2024<br>6 Predictors"]
      B2["Walk-Forward<br>Training Procedure"]
      B3["Algorithms: TD3 (DRL)<br>vs. Black-Scholes Delta"]
  end

  B --> C["Computational Process"]
  
  subgraph C ["Computational Process"]
      C1["Agent Training<br>Simulate Hedging Decisions"]
      C2["Constraints & Scenarios<br>Transaction Costs<br>Risk Penalties"]
  end

  C --> D["Key Findings & Outcomes"]
  
  subgraph D ["Key Findings & Outcomes"]
      D1["DRL Outperforms Delta<br>Specifically in Volatile/<br>High-Cost Environments"]
      D2["Performance Deteriorates<br>with High Risk-Awareness"]
      D3["Longer Volatility Estimation<br>Intervals = More Stable Results"]
  end