Paper: arXiv 2509.01590
Authors: Bingyang Wang, Grant Johnson, Maria Hybinette, Tucker Balch
Abstract
This paper investigates whether artificial intelligence can enhance stock clustering compared to traditional methods. We consider this in the context of the semi-strong Efficient Markets Hypothesis (EMH), which posits that prices fully reflect all public information and, accordingly, that clusters based on price information cannot be improved upon. We benchmark three clustering approaches: (i) price-based clusters derived from historical return correlations, (ii) human-informed clusters defined by the Global Industry Classification Standard (GICS), and (iii) AI-driven clusters constructed from large language model (LLM) embeddings of stock-related news headlines. At each date, each method provides a classification in which each stock is assigned to a cluster. To evaluate a clustering, we transform it into a synthetic factor model following the Arbitrage Pricing Theory (APT) framework. This enables consistent evaluation of predictive performance in a roll forward, out-of-sample test. Using S&P 500 constituents from from 2022 through 2024, we find that price-based clustering consistently outperforms both rule-based and AI-based methods, reducing root mean squared error (RMSE) by 15.9% relative to GICS and 14.7% relative to LLM embeddings. Our contributions are threefold: (i) a generalizable methodology that converts any equity grouping: manual, machine, or market-driven, into a real-time factor model for evaluation; (ii) the first direct comparison of price-based, human rule-based, and AI-based clustering under identical conditions; and (iii) empirical evidence reinforcing that short-horizon return information is largely contained in prices. These results support the EMH while offering practitioners a practical diagnostic for monitoring evolving sector structures and provide academics a framework for testing alternative hypotheses about how quickly markets absorb information.
Complexity vs Empirical Score
- Math Complexity: 6.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper uses advanced financial theory (APT, EMH) and embedding techniques, but the empirical setup is exceptionally rigorous with out-of-sample rolling tests, specific dataset details (S&P 500 2022-2024), and clear error metrics (RMSE).
Research Flowchart
flowchart TD
A["Research Question:<br>Can AI improve stock clustering<br>over price-based methods under EMH?"] --> B["Methodology Setup<br>Rolling Out-of-Sample Test"]
B --> C{"Inputs"}
C --> C1["S&P 500 Constituents<br>2022-2024"]
C --> C2["Three Clustering Methods<br>1. Price-Based<br>2. GICS (Human Rules)<br>3. LLM Embeddings"]
C --> C3["APT Factor Model Framework<br>Consistent Evaluation"]
C1 & C2 & C3 --> D["Computational Process<br>Convert clusters to synthetic factors<br>Calculate RMSE in out-of-sample rolls"]
D --> E{"Key Findings"}
E --> E1["Price-Based Clustering<br>Best Performance"]
E --> E2["15.9% RMSE Reduction<br>vs GICS"]
E --> E3["14.7% RMSE Reduction<br>vs LLM Embeddings"]
E --> E4["Conclusion:<br>Short-horizon returns<br>contained in prices (EMH supported)"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a generalizable methodology that converts any equity grouping into a real‑time synthetic factor model for out‑of‑sample evaluation.
Source sentence
a generalizable methodology that converts any equity grouping: manual, machine, or market-driven, into a real-time factor model for evaluation
Assumes 6
- Semi‑strong EMH holds, i.e., prices fully reflect all public information. market
Source sentence
The semi‑strong form of the Efficient Market Hypothesis (EMH) states that prices fully reflect all public information
- Number of clusters is fixed at K = 11 to match GICS sectors. model
Source sentence
we choose k = 11 because GICS is constrained to 11 sectors
- Linear factor loadings are estimated by OLS on one month of daily returns. model
Source sentence
At the end of each period, we estimate a linear model for each stock using one month of daily return data and Ordinary Least Squares regression
- Clusters are re‑computed weekly and predictions are made for the following week. execution
Source sentence
The linear model for each clustering is reevaluated weekly, and the errors are computed for each day rolling forward
Show 2 more
- Equal‑weighted average returns represent each cluster’s return series. model
Source sentence
For each cluster, we construct a daily return series by taking an equal-weighted average of the returns of all stocks in that cluster
- Back‑testing period starts 1 Jan 2022 due to the embedding knowledge‑cutoff. data
Source sentence
Our backtesting analysis is limited to the period beginning January 1, 2022, due to the September 2021 knowledge cutoff of OpenAI’s text-embedding-3-large embeddings
Data
- S&P 500 constituents market-prices commercial
Source sentence
Using S&P 500 constituents from from 2022 through 2024
- Historical price data market-prices commercial
Source sentence
Our price data is also sourced from WRDS
- GICS classification fundamentals commercial
Source sentence
We use point-in-time GICS classification data from the Compustat North America Daily Updates database
- News headline data text-news commercial
Source sentence
Our stock-level headline data is sourced from RavenPack
- LLM embeddings text-news proprietary
Source sentence
OpenAI’s text-embedding-3-large embeddings used in our
Costs
Not applicable
Finds 4
- Price‑based clustering reduces RMSE by 15.9% relative to GICS and 14.7% relative to LLM embeddings. performance
Source sentence
price-based clustering consistently outperforms both rule-based and AI-based methods, reducing root mean squared error (RMSE) by 15.9% relative to GICS and 14.7% relative to LLM embeddings
- The performance difference between the best returns‑based and best LLM‑based clustering is statistically significant (p‑value = 0). statistical
Source sentence
The resulting test yields a p-value of 0, indicating that the difference in performance between the two methods is statistically significant
- Human‑judged GICS and AI‑derived news embeddings do not provide superior predictive factors over correlation‑based clusters. negative
Source sentence
Human judgments regarding “economic similarity” and sophisticated NLP extractions of news text may be useful in other contexts, but they do not translate into superior predictive factors
- Dynamic correlation groupings are sufficient for portfolio construction and risk modeling. theoretical
Source sentence
For portfolio construction and risk modeling, relying on dynamically updated correlation groupings appears sufficient and essential to capture the cross‑sectional structure of systematic risk
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 34 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (134)
- Algorithm AS 136: A K-Means Clustering Algorithm (1979)
- Language Models are Few-Shot Learners (2020) — link
- Mistral 7B (2023) — link
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2018) — link
- Some methods for classification and analysis of multivariate observations (1967) — link
- Clustering by Means of Medoids (1987)
- A simple and fast algorithm for K-medoids clustering (2009) — link
- Cluster Analysis (2011) — link
and 126 more
- Ward’s Hierarchical Agglomerative Clustering Method: Which Algorithms Implement Ward’s Criterion? (2014) — link
- Hierarchical clustering schemes (1967) — link
- Fast and eager k-medoids clustering: O(k) runtime improvement of the PAM, CLARA, and CLARANS algorithms (2021) — link
- An empirical comparison of four initialization methods for the K-Means algorithm (1999) — link
- Initializing K-means Batch Clustering: A Critical Evaluation of Several Techniques (2007) — link
- Industry Classifications and Return Comovement (2007) — link
- Understanding trend inflation through the lens of the goods and services sectors (2023) — link
- Markets,Sectors, and the Trading Cube (2012) — link
- The Role of the S&P 500 Index Constituents in Tracking the U.S. Equity Market (2012) — link
- Nonlinear Interdependence Between the US and Emerging Markets’ Industrial Stock Sectors (2015) — link
- Financial clusters, industry groups, and stock return correlations (2021) — link
- What’s My Line? A Comparison of Industry Classification Schemes for Capital Market Research (2003) — link
- Understanding Defensive Equity (2014) — link
- Clustering in US Stock Prices after Decimalisation (2008) — link
- Correlation Matrix Clustering for Statistical Arbitrage Portfolios (2023) — link
- Clustering in US Stock Prices after Decimalisation (2008) — link
- Scope, Scale, and Concentration: The 21st-Century Firm (2025) — link
- Multi-Industry Simplex : A Probabilistic Extension of GICS — → 2310.04280
- Co-trading networks for modeling dynamic interdependency structures and estimating high-dimensional covariances in US equity markets — → 2302.09382
- A Comparison of Economic Agent-Based Model Calibration Methods (2019) — link
- Hierarchical structure in financial markets (1999)
- Global Industry Classification Standard (GICS\textsuperscript \textregistered ) Methodology (2024) — link
- Text-based network industries and endogenous product differentiation (2016)
- Efficient capital markets (1970)
- A Brief Survey on Anonymization Techniques for Privacy Preserving Publishing of Social Network Data (2008)
- Scalable Modeling of Real Graphs Using Kronecker Multiplication (2007)
- The Arbitrage Theory of Capital Asset Pricing (1976) — link
- Measurement-Calibrated Graph Models for Social Network Experiments (2010)
- Reverse stress testing interbank networks (2017)
- A survey on graph kernels (2020)
- Statistical mechanics of complex networks (2002)
- Statistical mechanics of networks (2004)
- On Random Graphs I (1959)
- JSON (2001) — link
- Fraud detection: A systematic literature review of graph-based anomaly detection approaches (2020)
- Using social network analysis to prevent money laundering (2017)
- Using the RetSim Fraud Simulation Tool to Set Thresholds for Triage of Retail Fraud (2015)
- Exploring Agent-Based Methods for the Analysis of Payment Systems: A Crisis Model for StarLogo TNG (2009)
- Survey of Approaches to Generate Realistic Synthetic Graphs (2016)
- Privacy and Anonymization in Social Networks (2014)
- Private Analysis of Graph Data (2014)
- Network models of financial systemic risk: A review (2017)
- Differentially Private Release of Synthetic Graphs (2020)
- Privacy-Preserving Dark Pools (2020)
- Informational Inequality: How High Frequency Traders Use Premier Access to Information to Prey on Institutional Investors (2016)
- Brokers and Order Flow Leakage: Evidence from Fire Sales (2019)
- An agent-based model of payment systems (2011)
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018)
- Reflections on Random Kitchen Sinks (2017) — link
- Enhancing trading strategies with order book signals (2018)
- Limit order books (2013)
- The European Union general data protection regulation: what it is and what it means (2019)
- Improved Techniques for Training GANs (2016)
- ABIDES: Towards High-Fidelity Market Simulation for AI Research (2019) — link
- Limit Order Books (2016)
- Trades, quotes and prices: financial markets under the microscope (2018)
- Evaluating Differentially Private Machine Learning in Practice (2019)
- How to Evaluate Trading Strategies: Single Agent Market Replay or Multiple Agent Interactive Simulation? (2019)
- Time Series Simulation by Conditional Generative Adversarial Net (2019) — link
- Generating Realistic Stock Market Order Streams (2019) — link
- Quant GANs: Deep Generation of Financial Time Series (2019) — link
- k-anonymity: a model for protecting privacy. International Journal on Uncertainty (2002)
- 50 Years of Data Science (2017)
- GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium (2017)
- Are GANs Created Equal? A Large-scale Study (2018)
- A note on the evaluation of generative models (2016) — link
- Are adversarial examples inevitable? (2019) — link
- Family Educational Rights and Privacy Act (FERPA) (2007)
- HIPAA and Protecting Health Information in the 21st Century (2018)
- What can we learn from the Facebook—Cambridge Analytica scandal? (2018)
- Enterprise data breach: causes, challenges, prevention, and future directions (2017)
- Financial market reaction to cyberattacks (2019)
- Protecting Privacy when Disclosing Information: $k$-Anonymity and Its Enforcement through Generalization and Suppression (1998)
- Calibrating Noise to Sensitivity in Private Data Analysis (2006)
- On $K$-anonymity and the Curse of Dimensionality (2005)
- Comparative Study of Differentially Private Data Synthesis Methods (2019) — link
- Privacy, Anonymity, and Big Data in the Social Sciences (2014)
- UCI Machine Learning Repository (2017) — link
- Assessing Generative Models via Precision and Recall (2018)
- Modeling Tabular data using Conditional GAN (2019) — link
- Airline Passenger Name Record Generation using Generative Adversarial Networks (2019) — link
- Generating Multi-label Discrete Patient Records using Generative Adversarial Networks (2017)
- Value at Risk: The New Benchmark for Managing Financial Risk (2006)
- A Review Of Synthetic Data Generation Methods For Privacy Preserving Data Publishing (2015)
- Privacy-preserving deep learning (2015)
- Differentially Private Mixture of Generative Neural Networks (2019)
- Generating Synthetic Data to Match Data Mining Patterns (2008)
- PrivBayes: Private Data Release via Bayesian Networks (2014)
- DPSynthesizer: Differentially Private Data Synthesizer for Privacy Preserving Data Sharing (2014)
- PeGS: Perturbed Gibbs Samplers That Generate Privacy-Compliant Synthetic Data (2014)
- Using Support Vector Machines for Generating Synthetic Datasets (2010)
- Random Forests for Generating Partially Synthetic, Categorical Data (2010)
- Differential Privacy in New Settings (2010)
- Differential Privacy Under Continual Observation (2010)
- Differentially Private Event Sequences over Infinite Streams (2014)
- Broadening the Scope of Differential Privacy Using Metrics (2013)
- Optstream: Releasing Time Series Privately (2019)
- Analysis of Financial Time Series (2010)
- Agent-based Computational Finance (2006)
- Who’s Watching? De-anonymization of Netflix Reviews using Amazon Reviews (2018)
- A Rigorous and Customizable Framework for Privacy (2012)
- The Synthetic Data Vault (2016)
- Protecting the 2020 Census (2019)
- A Kernel Two-sample Test (2012)
- Generating Multi-label Discrete Patient Records using Generative Adversarial Networks (2017)
- Data Synthesis based on Generative Adversarial Networks (2018)
- Modeling Tabular data using Conditional GAN (2019)
- A vector space model for automatic indexing (1975)
- PATE-GAN: Generating Synthetic Data with Differential Privacy Guarantees (2019) — link
- A Framework for Behavioural Cloning. (1995)
- Cluster-driven Hierarchical Representation of Large Asset Universes for Optimal Portfolio Construction (2024) — link
- Can Daily Closing Prices Predict Future Movements?\ The Role of Limit Order Clustering (2025) — link
- Fads, Martingales, and Market Efficiency (1990) — link
- Evidence of Predictable Behavior of Security Returns (1990) — link
- Artificial Intelligence for Measurement: Using Transformer-Based Embeddings to Cluster Firms by Economic Similarity (2024) — link
- Pairs Trading: Performance of a Relative‐Value Arbitrage Rule (2006) — link
- Correlation Matrix Clustering for Statistical Arbitrage Portfolios (2023) — link
- A Quantitative Approach to Tactical Asset Allocation (2007) — link
- Value and Momentum Everywhere (2013)
- The Properties of Equally Weighted Risk Contribution Portfolios (2010) — link
- Building Diversified Portfolios that Outperform Out of Sample (2016) — link
- Cluster analysis for portfolio optimization (2008)
- Portfolio Selection (1952)
- Compustat North America Daily Updates (2025)
- Wharton Research Data Services (WRDS) (2025)
- RavenPack News Analytics (2025)
2 of 134 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.