Paper: arXiv 2412.11462
Authors: Shasha Yu, Qinchen Zhang, Yuwei Zhao
Abstract
This project aims to predict short-term and long-term upward trends in the S&P 500 index using machine learning models and feature engineering based on the “101 Formulaic Alphas” methodology. The study employed multiple models, including Logistic Regression, Decision Trees, Random Forests, Neural Networks, K-Nearest Neighbors (KNN), and XGBoost, to identify market trends from historical stock data collected from Yahoo! Finance. Data preprocessing involved handling missing values, standardization, and iterative feature selection to ensure relevance and variability. For short-term predictions, KNN emerged as the most effective model, delivering robust performance with high recall for upward trends, while for long-term forecasts, XGBoost demonstrated the highest accuracy and AUC scores after hyperparameter tuning and class imbalance adjustments using SMOTE. Feature importance analysis highlighted the dominance of momentum-based and volume-related indicators in driving predictions. However, models exhibited limitations such as overfitting and low recall for positive market movements, particularly in imbalanced datasets. The study concludes that KNN is ideal for short-term alerts, whereas XGBoost is better suited for long-term trend forecasting. Future enhancements could include advanced architectures like Long Short-Term Memory (LSTM) networks and further feature refinement to improve precision and generalizability. These findings contribute to developing reliable machine learning tools for market trend prediction and investment decision-making.
Complexity vs Empirical Score
- Math Complexity: 4.0/10
- Empirical Rigor: 6.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper applies standardized feature engineering and common ML algorithms with cross-validation and hyperparameter tuning, indicating solid empirical work, but relies on established methodologies without deep mathematical derivation or novel theoretical proofs.
Research Flowchart
flowchart TD
A["Research Goal: Predict S&P 500 Trends"] --> B{"Data Collection & Processing"}
B --> C["Feature Engineering<br>101 Formulaic Alphas"]
C --> D{"Model Training & Tuning"}
D --> E["Short-Term Forecast"]
D --> F["Long-Term Forecast"]
E --> G["KNN<br>Best for Upward Trend Alerts"]
F --> H["XGBoost<br>Best Accuracy & AUC"]
G & H --> I["Findings:<br>Momentum & Volume Indicators Critical"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a machine‑learning based approach to predict short‑term and long‑term upward trends in the S&P 500 using feature engineering from the 101 Formulaic Alphas methodology.
Source sentence
This project aims to predict short-term and long-term upward trends in the S&P 500 index using machine learning models and feature engineering based on the "101 Formulaic Alphas" methodology.
Assumes 4
- Missing values are forward‑filled to maintain continuity of the time series. data
Source sentence
To prepare the data for analysis, missing values were forward-filled to maintain continuity and prevent gaps in the time series.
- Features are standardized to avoid bias from differing variable magnitudes. data
Source sentence
Standardization techniques were applied to ensure uniform scaling of features, thereby avoiding potential biases caused by differences in variable magnitudes.
- Class imbalance is addressed by applying SMOTE to rebalance the dataset for long‑term models. statistical
Source sentence
In the context of long-term predictions, the XGBoost model showed the greatest potential, particularly when applied to datasets balanced using the Synthetic Minority Oversampling Technique (SMOTE).
- A random seed of 42 ensures reproducibility of model training. other
Source sentence
To ensure reproducibility and robust evaluation, a random seed of 42 was used for all model training processes.
Data
- S&P 500 daily data market-prices public
Source sentence
The dataset used in this project was sourced from Yahoo! Finance and covered a timeframe from November 1, 2013, to October 31, 2024. The data included daily updates for the S&P 500 index and its 500 constituent stocks, capturing attributes such as Date, Open, High, Low, Close, Volume, and Adjusted Close.
Costs
Not applicable
Finds 4
- KNN achieved the best short‑term performance, delivering high recall for upward trends. predictive-accuracy
Source sentence
For short-term predictions, KNN emerged as the most effective model, delivering robust performance with high recall for upward trends
- XGBoost achieved the highest long‑term accuracy and AUC after hyper‑parameter tuning and SMOTE rebalancing. predictive-accuracy
Source sentence
for long-term forecasts, XGBoost demonstrated the highest accuracy and AUC scores after hyperparameter tuning and class imbalance adjustments using SMOTE
- Feature importance analysis highlighted momentum‑based and volume‑related indicators as the dominant predictors. statistical
Source sentence
Feature importance analysis highlighted the dominance of momentum-based and volume-related indicators in driving predictions
- All models exhibited limitations such as overfitting and low recall for positive market movements, especially in imbalanced datasets. negative
Source sentence
However, models exhibited limitations such as overfitting and low recall for positive market movements, particularly in imbalanced datasets
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 21 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.