Paper: arXiv 2512.23847

Authors: Zhenyu Gao, Wenxi Jiang, Yutong Yan

Abstract

We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs). Using state-of-the-art pre-training data detection techniques, we estimate the likelihood that a given prompt appeared in an LLM’s training corpus, a statistic we term Lookahead Propensity (LAP). We formally show that a positive correlation between LAP and forecast accuracy indicates the presence and magnitude of lookahead bias, and apply the test to two forecasting tasks: news headlines predicting stock returns and earnings call transcripts predicting capital expenditures. Our test provides a cost-efficient, diagnostic tool for assessing the validity and reliability of LLM-generated forecasts.

Complexity vs Empirical Score

  • Math Complexity: 4.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper develops a formal statistical test with econometric propositions but relies on established MIA metrics rather than novel, heavy derivations, placing it in the moderate math range. It demonstrates empirical rigor by applying the test to real financial datasets (news headlines and earnings calls) and validating with out-of-sample bootstrap analysis, though it lacks code or backtest-ready implementation details.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Test for Lookahead Bias<br>in LLM Economic Forecasts"] --> B{"Data & Input:<br>Financial Text Data"}
  B --> C["Process: Estimate<br>Lookahead Propensity LAP<br>via P(data in training corpus)"]
  C --> D{"Computation:<br>Correlation Analysis<br>LAP vs Forecast Accuracy"}
  D --> E{"Statistical Test:<br>Significant Positive Correlation?"}
  E -- Yes --> F["Outcome: Lookahead Bias<br>Detectable & Quantifiable"]
  E -- No --> G["Outcome: Bias Absent<br>or Insufficient Data"]