More Mentions, Less Information? Diagnosing Identity and Alignment in Reddit Embeddings for Volatility Forecasting

Paper: arXiv 2610.25468

Authors: Zhiyu Zheng

Abstract

Since the 2021 GameStop short squeeze, r/WallStreetBets (WSB) has been associated with episodes of extreme volatility, and its text is increasingly used to forecast volatility. A model using only large language model (LLM) embeddings of WSB posts attains an out-of-sample $R^2$ of 0.13 for log realized variance. That number shows predictive power, but it does not reveal where the information comes from or how much the text adds beyond a firm’s own volatility history. We answer these questions with a diagnostic protocol that nests embeddings on a heterogeneous autoregressive (HAR) benchmark, compares them with text-free firm-identity controls, shuffles the day-to-text pairing within firm-years to test day-specific alignment, and checks whether gains persist across periods and coverage groups. We apply the protocol to frozen LLM embeddings in a panel of 202,145 mentioned stock-days, using retrospective rolling out-of-sample tests over 2015-2022. The results indicate that most of the standalone fit reflects firm or firm-year information: on firms seen in training, a text-free firm baseline reaches $R^2 = 0.32$, shuffling preserves 89-92% of the standalone fit, and masking ticker strings reduces standalone $R^2$ to -0.005. Adding embeddings to HAR reduces equal-date MSE by 1.22% (95% CI [0.94, 1.49]). This small but statistically significant gain survives firm controls and ticker masking, but is largely eliminated by alignment shuffling. The gain is concentrated in 2015-2019 and is not distinguishable from zero in 2020-2022. Expanding coverage by 48% through off-hours and weekend posts adds stock-days with no reliable positive increment. More mentions therefore do not imply more incremental forecasting information.

Complexity vs Empirical Score

• Math Complexity: 6.0/10
• Empirical Rigor: 9.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a highly rigorous empirical study with a novel diagnostic protocol to assess the true incremental value of LLM embeddings in volatility forecasting. While the mathematical complexity is moderate, the statistical methodology and extensive out-of-sample testing are exceptionally strong, leading to significant and well-supported conclusions.

Research Flowchart

  flowchart TD
    A[Research Goal: Diagnose Information in WSB Embeddings for Volatility Forecasting] --> B{Key Methodology Steps};
    B --> C[Data/Inputs: 202,145 Mentioned Stock-Days (WSB Posts 2015-2022), Frozen LLM Embeddings];
    C --> D[Computational Processes: Nesting on HAR Benchmark, Firm-Identity Controls, Day-to-Text Alignment Shuffling, Retrospective Rolling OOS Tests];
    D --> E[Key Findings: Most standalone fit is firm/firm-year info; Small (1.22% MSE reduction) but significant gain from embeddings on HAR, eliminated by alignment shuffling; Gain concentrated 2015-2019, zero 2020-2022; More mentions don't imply more incremental info.];

Cited References (33)

1. Financial Regulation and AI: A Faustian Bargain? (2025) — link
2. The Llama 3 Herd of Models (2024) — link
3. Place Your Bets? The Value of Investment Research on Reddit’s Wallstreetbets (2023) — link
4. FinBERT : A Large Language Model for Extracting Information from Financial Text† (2022) — link
5. Realised Volatility Forecasting: Machine Learning via Financial Word Embedding (2021) — link
6. Probing Classifiers: Promises, Shortcomings, and Advances (2021) — link
7. The Pushshift Reddit Dataset (2020) — link
8. Why Don’t We Agree? Evidence from a Social Network of Investors (2019) — link

0 of 33 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.