Paper: arXiv 2412.11462

Authors: Shasha Yu, Qinchen Zhang, Yuwei Zhao

Abstract

This project aims to predict short-term and long-term upward trends in the S&P 500 index using machine learning models and feature engineering based on the “101 Formulaic Alphas” methodology. The study employed multiple models, including Logistic Regression, Decision Trees, Random Forests, Neural Networks, K-Nearest Neighbors (KNN), and XGBoost, to identify market trends from historical stock data collected from Yahoo! Finance. Data preprocessing involved handling missing values, standardization, and iterative feature selection to ensure relevance and variability. For short-term predictions, KNN emerged as the most effective model, delivering robust performance with high recall for upward trends, while for long-term forecasts, XGBoost demonstrated the highest accuracy and AUC scores after hyperparameter tuning and class imbalance adjustments using SMOTE. Feature importance analysis highlighted the dominance of momentum-based and volume-related indicators in driving predictions. However, models exhibited limitations such as overfitting and low recall for positive market movements, particularly in imbalanced datasets. The study concludes that KNN is ideal for short-term alerts, whereas XGBoost is better suited for long-term trend forecasting. Future enhancements could include advanced architectures like Long Short-Term Memory (LSTM) networks and further feature refinement to improve precision and generalizability. These findings contribute to developing reliable machine learning tools for market trend prediction and investment decision-making.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 6.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper applies standardized feature engineering and common ML algorithms with cross-validation and hyperparameter tuning, indicating solid empirical work, but relies on established methodologies without deep mathematical derivation or novel theoretical proofs.

Research Flowchart

  flowchart TD
  A["Research Goal: Predict S&P 500 Trends"] --> B{"Data Collection & Processing"}
  B --> C["Feature Engineering<br>101 Formulaic Alphas"]
  C --> D{"Model Training & Tuning"}
  D --> E["Short-Term Forecast"]
  D --> F["Long-Term Forecast"]
  E --> G["KNN<br>Best for Upward Trend Alerts"]
  F --> H["XGBoost<br>Best Accuracy & AUC"]
  G & H --> I["Findings:<br>Momentum & Volume Indicators Critical"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a machine‑learning based approach to predict short‑term and long‑term upward trends in the S&P 500 using feature engineering from the 101 Formulaic Alphas methodology.

Source sentence
This project aims to predict short-term and long-term upward trends in the S&P 500 index using machine learning models and feature engineering based on the "101 Formulaic Alphas" methodology.

Assumes 4

  • Missing values are forward‑filled to maintain continuity of the time series. data
    Source sentence
    To prepare the data for analysis, missing values were forward-filled to maintain continuity and prevent gaps in the time series.
  • Features are standardized to avoid bias from differing variable magnitudes. data
    Source sentence
    Standardization techniques were applied to ensure uniform scaling of features, thereby avoiding potential biases caused by differences in variable magnitudes.
  • Class imbalance is addressed by applying SMOTE to rebalance the dataset for long‑term models. statistical
    Source sentence
    In the context of long-term predictions, the XGBoost model showed the greatest potential, particularly when applied to datasets balanced using the Synthetic Minority Oversampling Technique (SMOTE).
  • A random seed of 42 ensures reproducibility of model training. other
    Source sentence
    To ensure reproducibility and robust evaluation, a random seed of 42 was used for all model training processes.

Data

  • S&P 500 daily data · Yahoo! Finance market-prices public
    November 1, 2013 to October 31, 2024 · S&P 500 index and its 500 constituent stocks
    Source sentence
    The dataset used in this project was sourced from Yahoo! Finance and covered a timeframe from November 1, 2013, to October 31, 2024. The data included daily updates for the S&P 500 index and its 500 constituent stocks, capturing attributes such as Date, Open, High, Low, Close, Volume, and Adjusted Close.

Recognised sources: Yahoo Finance

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • KNN achieved the best short‑term performance, delivering high recall for upward trends. predictive-accuracy
    Source sentence
    For short-term predictions, KNN emerged as the most effective model, delivering robust performance with high recall for upward trends
  • XGBoost achieved the highest long‑term accuracy and AUC after hyper‑parameter tuning and SMOTE rebalancing. predictive-accuracy
    Source sentence
    for long-term forecasts, XGBoost demonstrated the highest accuracy and AUC scores after hyperparameter tuning and class imbalance adjustments using SMOTE
  • Feature importance analysis highlighted momentum‑based and volume‑related indicators as the dominant predictors. statistical
    Source sentence
    Feature importance analysis highlighted the dominance of momentum-based and volume-related indicators in driving predictions
  • All models exhibited limitations such as overfitting and low recall for positive market movements, especially in imbalanced datasets. negative
    Source sentence
    However, models exhibited limitations such as overfitting and low recall for positive market movements, particularly in imbalanced datasets

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 21 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.