Application of Deep Reinforcement Learning to At-the-Money S&P 500 Options Hedging

ArXiv ID: 2510.09247 “View on arXiv”

Authors: Zofia Bracha, Paweł Sakowski, Jakub Michańków

Abstract

This paper explores the application of deep Q-learning to hedging at-the-money options on the S&P500 index. We develop an agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, trained to simulate hedging decisions without making explicit model assumptions on price dynamics. The agent was trained on historical intraday prices of S&P500 call options across years 2004–2024, using a single time series of six predictor variables: option price, underlying asset price, moneyness, time to maturity, realized volatility, and current hedge position. A walk-forward procedure was applied for training, which led to nearly 17~years of out-of-sample evaluation. The performance of the deep reinforcement learning (DRL) agent is benchmarked against the Black–Scholes delta-hedging strategy over the same period. We assess both approaches using metrics such as annualized return, volatility, information ratio, and Sharpe ratio. To test the models’ adaptability, we performed simulations across varying market conditions and added constraints such as transaction costs and risk-awareness penalties. Our results show that the DRL agent can outperform traditional hedging methods, particularly in volatile or high-cost environments, highlighting its robustness and flexibility in practical trading contexts. While the agent consistently outperforms delta-hedging, its performance deteriorates when the risk-awareness parameter is higher. We also observed that the longer the time interval used for volatility estimation, the more stable the results.

Keywords: Deep Q-learning, Twin Delayed Deep Deterministic Policy Gradient (TD3), Delta hedging, Deep Reinforcement Learning (DRL), Walk-forward procedure, Equity Derivatives (S&P 500 Options)

Complexity vs Empirical Score

• Math Complexity: 8.5/10
• Empirical Rigor: 7.5/10
• Quadrant: Holy Grail
• Why: The paper employs advanced deep reinforcement learning algorithms like TD3 and discusses underlying theory (Bellman equations, Q-learning), contributing to high math complexity. Empirical rigor is strong due to extensive historical intraday data (2004-2024), walk-forward out-of-sample testing, multiple performance metrics, and sensitivity analyses on transaction costs and volatility windows.

  flowchart TD
    A["Research Goal<br>Apply Deep RL to<br>ATM S&P 500 Options Hedging"] --> B["Data & Methodology"]
    
    subgraph B ["Data & Methodology"]
        B1["Historical Data<br>2004-2024<br>6 Predictors"]
        B2["Walk-Forward<br>Training Procedure"]
        B3["Algorithms: TD3 (DRL)<br>vs. Black-Scholes Delta"]
    end

    B --> C["Computational Process"]
    
    subgraph C ["Computational Process"]
        C1["Agent Training<br>Simulate Hedging Decisions"]
        C2["Constraints & Scenarios<br>Transaction Costs<br>Risk Penalties"]
    end

    C --> D["Key Findings & Outcomes"]
    
    subgraph D ["Key Findings & Outcomes"]
        D1["DRL Outperforms Delta<br>Specifically in Volatile/<br>High-Cost Environments"]
        D2["Performance Deteriorates<br>with High Risk-Awareness"]
        D3["Longer Volatility Estimation<br>Intervals = More Stable Results"]
    end