Paper: arXiv 2412.12539

Authors: Vidhi Agrawal, Eesha Khalid, Tianyu Tan, Doris Xu

Abstract

This study applies machine learning to predict S&P 500 membership changes: key events that profoundly impact investor behavior and market dynamics. Quarterly data from WRDS datasets (2013 onwards) was used, incorporating features such as industry classification, financial data, market data, and corporate governance indicators. Using a Random Forest model, we achieved a test F1 score of 0.85, outperforming logistic regression and SVC models. This research not only showcases the power of machine learning for financial forecasting but also emphasizes model transparency through SHAP analysis and feature engineering. The model’s real world applicability is demonstrated with predicted changes for Q3 2023, such as the addition of Uber (UBER) and the removal of SolarEdge Technologies (SEDG). By incorporating these predictions into a trading strategy i.e. buying stocks announced for addition and shorting those marked for removal, we anticipate capturing alpha and enhancing investment decision making, offering valuable insights into index dynamics

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper uses standard machine learning models (Random Forest) and basic statistical analysis without advanced mathematical derivations, while it is heavily data-driven with specific datasets (WRDS, CRSP), detailed feature engineering, and attempts to validate predictions with actual trading strategy outcomes.

Research Flowchart

  flowchart TD
  A["Research Goal: Predict S&P 500 Membership Changes"] --> B["Data Collection: WRDS Datasets"]
  B --> C["Feature Engineering: Financial, Market, & Governance Indicators"]
  C --> D["Modeling: Random Forest Classifier"]
  D --> E["Validation: Test F1 Score 0.85"]
  E --> F["Interpretability: SHAP Analysis"]
  F --> G["Outcomes: Real-world Predictions & Trading Strategy"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a machine‑learning model to forecast S&P 500 additions and deletions and a trading strategy that buys announced additions and shorts announced removals.

Source sentence
By incorporating these predictions into a trading strategy i.e. buying stocks announced for addition and shorting those marked for removal, we anticipate capturing alpha and enhancing investment decision making

Assumes 1

  • Missing financial metric values are filled by forward‑filling the rolling mean. data
    Source sentence
    Missing values were imputed using a forward-fill technique, where the missing value was replaced with the rolling mean of the p…

Data

  • CRSP Data · WRDS market-prices commercial
    2013-2023 · All firms with daily market‑cap values, top 800 by market cap each quarter
    Source sentence
    The data was filtered to include the last trading day of each quarter. Firms were ranked by market capitalization, and the top 800 firms for each quarter were retained for further analysis.
  • Compustat Fundamentals · WRDS fundamentals commercial
    2013-2023 · All firms in the CRSP sample
    Source sentence
    Financial statement data from Compustat was integrated to capture firm-level financial indicators, including: Total assets. total liabilities, net income, operating income, cash flow from operations, ratio of current assets to current liabilities, ratio of total liabilities to equity, return on assets, return on equity
  • IBES Data · WRDS alternative commercial
    2013-2023 · All firms in the CRSP sample
    Source sentence
    Analyst coverage data from IBES was utilized, focusing on the number of analysts providing earnings estimates for each firm. This variable (num_analysts_covering) provides insight into the level of market attention a firm receives.
  • Audit Analytics · WRDS alternative commercial
    2013-2023 · All firms in the CRSP sample
    Source sentence
    Data on auditor changes and financial restatements was sourced from Audit Analytics: Number of auditor changes for each firm and number of financial restatements.
  • S&P 500 Membership Data · WRDS (CRSP dsp500 table) benchmark commercial
    2013-2023 · S&P 500 constituents and candidates
    Source sentence
    Data on S&P 500 membership was retrieved from CRSP's dsp500 table. This dataset includes the inclusion and exclusion dates for each firm in the S&P 500. This information was merged with quarterly data, creating a binary label: in_sp500.

Recognised sources: WRDS CRSP Compustat I/B/E/S

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • Random Forest achieved a test F1‑score of 0.85, outperforming logistic regression (0.70) and SVC models. predictive-accuracy
    Source sentence
    Random Forest turned out to be the best model in terms of performance, with a test F1-score of 0.85 and strong cross-validation results using a TimeSeriesSplit approach.
  • SHAP analysis identified 3‑month average trading volume, number of analysts covering, and operating income as key predictors of inclusion. statistical
    Source sentence
    The model identified key predictors for inclusion, such as the 3-month average trading volume, number of analysts covering the stock, operating incom…
  • The model correctly predicted the addition of Uber Technologies and the removal of SolarEdge Technologies for 2023. performance
    Source sentence
    In terms of actual prediction for test data for 2023, we predicted the addition of Uber Technologies (UBER) and Jabil (JBL) starting the 3rd quarter but the actual addition happened in the 4th quarter.
  • Cross‑validation using TimeSeriesSplit (K=5) yielded an out‑of‑sample test F1‑score of 85%. predictive-accuracy
    Source sentence
    of 87% and an out-of-sample test F1-score of 85%. To ensure robustness of the results, we performed cross-validation using TimeSeriesSplit with a K-fold value of 5

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 27 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.