Paper: arXiv 2510.03236

Authors: Ava C. Blake, Nivika A. Gandhi, Anurag R. Jakkula

Abstract

Accurate prediction of financial market volatility is critical for risk management, derivatives pricing, and investment strategy. In this study, we propose a multitude of regime-switching methods to improve the prediction of S&P 500 volatility by capturing structural changes in the market across time. We use eleven years of SPX data, from May 1st, 2014 to May 27th, 2025, to compute daily realized volatility (RV) from 5-minute intraday log returns, adjusted for irregular trading days. To enhance forecast accuracy, we engineered features to capture both historical dynamics and forward-looking market sentiment across regimes. The regime-switching methods include a soft Markov switching algorithm to estimate soft-regime probabilities, a distributional spectral clustering method that uses XGBoost to assign clusters at prediction time, and a coefficient-based soft regime algorithm that extracts HAR coefficients from time segments segmented through the Mood test and clusters through Bayesian GMM for soft regime weights, using XGBoost to predict regime probabilities. Models were evaluated across three time periods–before, during, and after the COVID-19 pandemic. The coefficient-based clustering algorithm outperformed all other models, including the baseline autoregressive model, during all time periods. Additionally, each model was evaluated on its recursive forecasting performance for 5- and 10-day horizons during each time period. The findings of this study demonstrate the value of regime-aware modeling frameworks and soft clustering approaches in improving volatility forecasting, especially during periods of heightened uncertainty and structural change.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced econometric methods including Markov switching, distributional spectral clustering, Bayesian GMM, and XGBoost, indicating high mathematical density. It uses substantial real-world data (11 years of 5-minute S&P 500 data) with detailed preprocessing, out-of-sample backtesting across distinct market regimes, and recursive forecasting, demonstrating strong empirical implementation.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Improve S&P 500 Volatility<br>Forecasting via Regime Switching"] --> B{"Data Acquisition & Feature Engineering"}
  B --> C["Methodology<br>Regime-Switching Algorithms"]
  
  subgraph B ["Data & Features"]
      B1["SPX Data<br>2014-2025"]
      B2["Daily Realized Volatility RV"]
      B3["Historical & Sentiment Features"]
  end

  subgraph C ["Core Methods"]
      C1["Soft Markov Switching"]
      C2["Spectral Clustering + XGBoost"]
      C3["Coefficient-Based + Bayesian GMM + XGBoost"]
  end

  C --> D{"Model Evaluation"}
  
  subgraph D ["Evaluation Framework"]
      D1["Time Periods<br>Pre/During/Post COVID"]
      D2["Horizons<br>1, 5, 10 Days"]
      D3["Baseline: Autoregressive Model"]
  end

  D --> E["Key Findings"]
  
  subgraph E ["Outcomes"]
      E1["Coefficient-Based Algorithm<br>Best Overall Performance"]
      E2["Outperformed Baseline &<br>All Other Regime Methods"]
      E3["High Value in<br>Uncertain/Regime-Shifting Periods"]
  end

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes multiple regime‑switching methods, notably a coefficient‑based clustering approach, to improve S&P 500 volatility forecasting.

Source sentence
In this study, we propose a multitude of regime-switching methods to improve the prediction of S&P 500 volatility by capturing structural changes in the market across time.

Assumes 5

  • The 5‑minute SPX closing prices are accurate and free of market microstructure noise. data
    Source sentence
    It is assumed that the 5-minute SPX closing prices are accurate and free of market microstructure noise, such as bid-ask bounce or discreteness of price changes.
  • The closing price in each interval is a fair consolidated measurement of market activity, ignoring closing anomalies. data
    Source sentence
    We also treat the closing price in each interval as a fair and consolidated measurement of market activity, ignoring any possible closing anomalies.
  • HAR model residuals are homoskedastic, independent, and normally distributed with zero mean. statistical
    Source sentence
    The HAR model extends a linear regression framework to forecast realized volatility and thus inherits the core assumptions of linear regression: homoskedasticity, implying constant variance of the residuals, that residuals are independent, and that residuals are normally distributed with a mean of zero.
  • Volatility follows different statistical behaviors (regimes) over time and regime transitions follow a Markov process. model
    Source sentence
    In order to utilize a regime-switching Markov model, we assume the following: volatility follows different statistical behaviors (regimes) at different times and that the transitions between these regimes follow a Markov process - the probability of transitioning to the next regime depends only on the present regime.
Show 1 more
  • Market regimes can be inferred from observable data. model
    Source sentence
    More broadly, we assume that these market regimes can be inferred from observable data.

Data

  • SPX 5‑minute intraday price data · Bloomberg Terminal market-prices commercial
    May 1st, 2014 to May 27th, 2025 · S&P 500 Index
    Source sentence
    Our dataset comprises intraday price data in 5‑minute intervals from the Standard & Poor’s 500 (or S&P 500) Index. ... The

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • The coefficient‑based clustering model achieved the lowest MSE across all periods, improving over the baseline HAR by 8.52% pre‑COVID, 9.73% during COVID, and 11.9% post‑COVID. predictive-accuracy
    Source sentence
    It consistently reduced MSE compared to the standard HAR model, with improvements of 8.52% (Pre-COVID), 9.73% (COVID), and 11.9% (Post-COVID).
  • All regime‑switching models generally outperformed the baseline HAR model, with the Markov regime‑switching model close behind in pre‑ and post‑COVID periods. performance
    Source sentence
    Across all periods, regime-switching models generally outperformed the baseline HAR model. In particular, the Coefficient-Based Clustering model achieved the lowest MSE during all time periods.
  • The Markov HAR model struggled during the highly volatile COVID period, performing marginally worse than the standard non‑regime model. negative
    Source sentence
    The highly volatile COVID time period proved to be more difficult for the Markov HAR to predict, and the standard model was marginally more accurate.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 30 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (21)

  1. Forecasting the Chinese stock market volatility with international market volatilities: The role of regime switching (2020)
  2. Forecasting realized volatility of agricultural commodity futures with infinite Hidden Markov HAR models (2022)
  3. Forecasting realized volatility with changing average levels (2020)
  4. Forecasting realised volatility using regime-switching models (2025)
  5. Stock Price Volatility Prediction with Long Short-Term Memory Neural Networks (2018)
  6. Realized Volatility Forecasting with Neural Networks (2019)
  7. Forecasting the volatility of educational firms based on HAR model and LSTM models considering sentiment and educational policy (2024)
  8. Forecasting S&P 500 Volatility with the HAR-RV Model (2024) — link
and 13 more
  1. A Practical Guide to harnessing the HAR volatility model (2021)
  2. A novel HAR-type realized volatility forecasting model using graph neural network (2025)
  3. A Simple Approximate Long-Memory Model of Realized Volatility (2009)
  4. Why Volatility Is Important for Investors (2022) — link
  5. Prediction of realized volatility and implied volatility indices using AI and machine learning: A review (2024)
  6. A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle (1989)
  7. Structural Clustering of Volatility Regimes for Dynamic Trading Strategies (2021) — link
  8. Regime switching model estimation: spectral clustering hidden Markov model (2019)
  9. Hybrid Volatility Forecasting Models Based on Machine Learning of High-Frequency Data (2022)
  10. Forecasting volatility of China’s crude oil futures based on hybrid ML-HAR-RV models (2025) — link
  11. A Nonparametric Bayesian Approach to Detect the Number of Regimes in Markov Switching Models (2007)
  12. Spectral Clustering and Embedding with Hidden Markov Models (2007)
  13. Kenneth R.\ French Data Library (2025)

0 of 21 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.