{"paper_id":"2412.12539","title":"Hunting Tomorrow's Leaders: Using Machine Learning to Forecast S&P 500 Additions & Removal","authors":["Vidhi Agrawal","Eesha Khalid","Tianyu Tan","Doris Xu"],"paper_date":"2024-12-17","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2412.12539v1","page_url":"https://thequant.space/flowcharts/hunting-tomorrows-leaders-using-machine-learning-to-foreca/","scores":{"empirical_rigor":7.5,"math_complexity":2.5,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes a machine‑learning model to forecast S&P 500 additions and deletions and a trading strategy that buys announced additions and shorts announced removals.","quote":"By incorporating these predictions into a trading strategy i.e. buying stocks announced for addition and shorting those marked for removal, we anticipate capturing alpha and enhancing investment decision making","summary":"This study applies machine learning to predict S&P 500 membership changes: key events that profoundly impact investor behavior and market dynamics. Quarterly data from WRDS datasets (2013 onwards) was used, incorporating features such as industry classification, financial data, market data, and…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Missing financial metric values are filled by forward‑filling the rolling mean.","type":"data","stated":"explicit","quote":"Missing values were imputed using a forward-fill technique, where the missing value was replaced with the rolling mean of the p…"}]},"datasets":{"status":"available","items":[{"name":"CRSP Data","provider":"WRDS","kind":"market-prices","access":"commercial","period":"2013-2023","universe":"All firms with daily market‑cap values, top 800 by market cap each quarter","quote":"The data was filtered to include the last trading day of each quarter. Firms were ranked by market capitalization, and the top 800 firms for each quarter were retained for further analysis.","known_id":"crsp"},{"name":"Compustat Fundamentals","provider":"WRDS","kind":"fundamentals","access":"commercial","period":"2013-2023","universe":"All firms in the CRSP sample","quote":"Financial statement data from Compustat was integrated to capture firm-level financial indicators, including: Total assets. total liabilities, net income, operating income, cash flow from operations, ratio of current assets to current liabilities, ratio of total liabilities to equity, return on assets, return on equity","known_id":"compustat"},{"name":"IBES Data","provider":"WRDS","kind":"alternative","access":"commercial","period":"2013-2023","universe":"All firms in the CRSP sample","quote":"Analyst coverage data from IBES was utilized, focusing on the number of analysts providing earnings estimates for each firm. This variable (num_analysts_covering) provides insight into the level of market attention a firm receives.","known_id":"wrds"},{"name":"Audit Analytics","provider":"WRDS","kind":"alternative","access":"commercial","period":"2013-2023","universe":"All firms in the CRSP sample","quote":"Data on auditor changes and financial restatements was sourced from Audit Analytics: Number of auditor changes for each firm and number of financial restatements.","known_id":"wrds"},{"name":"S&P 500 Membership Data","provider":"WRDS (CRSP dsp500 table)","kind":"benchmark","access":"commercial","period":"2013-2023","universe":"S&P 500 constituents and candidates","quote":"Data on S&P 500 membership was retrieved from CRSP's dsp500 table. This dataset includes the inclusion and exclusion dates for each firm in the S&P 500. This information was merged with quarterly data, creating a binary label: in_sp500.","known_id":"crsp"}],"known":[{"id":"wrds","name":"WRDS","provider":"Wharton Research Data Services","kind":"other","access":"commercial","quote":"DATA A. Data Sources and Extraction Process The dataset used for this study was constructed by combining data from multiple sources, specifically the Wharton Research Data Services (WRDS) platform.","mentions":7,"used_in":3},{"id":"crsp","name":"CRSP","provider":"Center for Research in Security Prices","kind":"market-prices","access":"commercial","quote":"Between December 13, 2019, and September 24, 2024, we analyzed the effects of S&P 500 additions and deletions using announcements from S&P's website and data from CRSP.","mentions":5,"used_in":3},{"id":"compustat","name":"Compustat","provider":"S&P Global","kind":"fundamentals","access":"commercial","quote":"However, integrating monthly data from Compustat with S&P data resulted in approximately 85% missing values due to noise in the WRDS dataset.","mentions":4,"used_in":2},{"id":"ibes","name":"I/B/E/S","provider":"LSEG","kind":"fundamentals","access":"commercial","quote":"IBES Data: Analyst coverage data from IBES was utilized, focusing on the number of analysts providing earnings estimates for each firm.","mentions":2,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"weekly-monthly"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"n/a","evidence_card":"no","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"Random Forest achieved a test F1‑score of 0.85, outperforming logistic regression (0.70) and SVC models.","kind":"predictive-accuracy","quote":"Random Forest turned out to be the best model in terms of performance, with a test F1-score of 0.85 and strong cross-validation results using a TimeSeriesSplit approach."},{"finding":"SHAP analysis identified 3‑month average trading volume, number of analysts covering, and operating income as key predictors of inclusion.","kind":"statistical","quote":"The model identified key predictors for inclusion, such as the 3-month average trading volume, number of analysts covering the stock, operating incom…"},{"finding":"The model correctly predicted the addition of Uber Technologies and the removal of SolarEdge Technologies for 2023.","kind":"performance","quote":"In terms of actual prediction for test data for 2023, we predicted the addition of Uber Technologies (UBER) and Jabil (JBL) starting the 3rd quarter but the actual addition happened in the 4th quarter."},{"finding":"Cross‑validation using TimeSeriesSplit (K=5) yielded an out‑of‑sample test F1‑score of 85%.","kind":"predictive-accuracy","quote":"of 87% and an out-of-sample test F1-score of 85%. To ensure robustness of the results, we performed cross-validation using TimeSeriesSplit with a K-fold value of 5"}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":27,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:49:29Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}