Paper: arXiv 2301.00248

Authors: Thomas Dierckx, Jesse Davis, Wim Schoutens

Abstract

In this study, we predict next-day movements of stock end-of-day implied volatility using random forests. Through an ablation study, we examine the usefulness of different sources of predictors and expose the value of attention and sentiment features extracted from Twitter. We study the approach on a stock universe comprised of the 165 most liquid US stocks diversified across the 11 traditional market sectors using a sizeable out-of-sample period spanning over six years. In doing so, we uncover that stocks in certain sectors, such as Consumer Discretionary, Technology, Real Estate, and Utilities are easier to predict than others. Further analysis shows that possible reasons for these discrepancies might be caused by either excess social media attention or low option liquidity. Lastly, we explore how our proposed approach fares throughout time by identifying four underlying market regimes in implied volatility using hidden Markov models. We find that most added value is achieved in regimes associated with lower implied volatility, but optimal regimes vary per market sector.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper relies on standard applied machine learning techniques (Random Forests, Hidden Markov Models) rather than deep theoretical mathematics, resulting in lower math complexity. However, it demonstrates strong empirical rigor with a robust 6-year out-of-sample test across 165 stocks, offering novel insights into sector-specific volatility nowcasting using social media features.

Research Flowchart

  flowchart TD
    A["Research Goal:<br>Predict Next-Day Stock Implied Volatility (IV)"] --> B["Data Collection:<br>165 US Stocks across 11 Sectors<br>Twitter Sentiment & Attention + Option Data"]
    B --> C["Random Forest Modeling:<br>Ablation Study on Predictor Sources"]
    B --> D["Hidden Markov Models:<br>Identify 4 IV Market Regimes"]
    C & D --> E["Sector Analysis:<br>High predictability in Tech, Consumer Disc., Real Estate & Utilities"]
    C & D --> F["Regime Analysis:<br>Highest added value in low-IV regimes"]
    E & F --> G["Key Outcomes:<br>Twitter features improve predictions; discrepancies caused by social media attention & option liquidity"]