Paper: arXiv 2512.23847

Authors: Zhenyu Gao, Wenxi Jiang, Yutong Yan

Abstract

We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs). Using state-of-the-art pre-training data detection techniques, we estimate the likelihood that a given prompt appeared in an LLM’s training corpus, a statistic we term Lookahead Propensity (LAP). We formally show that a positive correlation between LAP and forecast accuracy indicates the presence and magnitude of lookahead bias, and apply the test to two forecasting tasks: news headlines predicting stock returns and earnings call transcripts predicting capital expenditures. Our test provides a cost-efficient, diagnostic tool for assessing the validity and reliability of LLM-generated forecasts.

Complexity vs Empirical Score

  • Math Complexity: 4.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper develops a formal statistical test with econometric propositions but relies on established MIA metrics rather than novel, heavy derivations, placing it in the moderate math range. It demonstrates empirical rigor by applying the test to real financial datasets (news headlines and earnings calls) and validating with out-of-sample bootstrap analysis, though it lacks code or backtest-ready implementation details.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Test for Lookahead Bias<br>in LLM Economic Forecasts"] --> B{"Data & Input:<br>Financial Text Data"}
  B --> C["Process: Estimate<br>Lookahead Propensity LAP<br>via P(data in training corpus)"]
  C --> D{"Computation:<br>Correlation Analysis<br>LAP vs Forecast Accuracy"}
  D --> E{"Statistical Test:<br>Significant Positive Correlation?"}
  E -- Yes --> F["Outcome: Lookahead Bias<br>Detectable & Quantifiable"]
  E -- No --> G["Outcome: Bias Absent<br>or Insufficient Data"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a statistical test based on Lookahead Propensity (LAP) to detect lookahead bias in LLM-generated economic forecasts.

Source sentence
We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs).

Assumes 4

  • A positive correlation between LAP and forecast accuracy indicates lookahead bias rather than genuine predictive reasoning. statistical
    Source sentence
    The main proposition implies that if forecast accuracy is correlated with LAP, such a relationship indicates lookahead bias rather than genuine predictive reasoning.
  • Headlines published before 4 p.m. are linked to the next‑day stock return; otherwise they are treated as news on day t + 1. execution
    Source sentence
    For simplicity, we use stock market close time 4 p.m. as the cutoff to classify headlines on day t.
  • Llama‑3.3 provides access to prompt token probabilities needed to compute LAP. model
    Source sentence
    Most importantly, Llama-3.3 provides access to prompt token probabilities, which are essential for computing our LAP measure.
  • Out‑of‑sample period starts at the model’s release date rather than its knowledge cutoff. other
    Source sentence
    To be conservative, we use the model’s release date—rather than its claimed knowledge cutoff—as the starting point for our out-of-sample

Data

  • CRSP daily stock file · CRSP market-prices commercial
    all common stocks listed on NYSE, NASDAQ · common stocks with share code 10 or 11
    Source sentence
    on the New York Stock Exchange (NYSE), the National Association of Securities Dealers Automated Quotations (NASDAQ), and the American Stock Exchange (AMEX).
  • Bloomberg News headlines · Bloomberg L.P. text-news commercial
    January 2012 to December 2023 · 91,361 news headlines for 1,587 unique companies
    Source sentence
    The final dataset consists of 91,361 news headlines covering 1,587 unique companies from January 2012 to December 2023.
  • Thomson Reuters StreetEvents earnings call transcripts · Thomson Reuters text-news commercial
    2006 to 2020 · 74,338 firm‑quarter observations from 3,897 unique U.S. publicly listed firms
    Source sentence
    our final sample consists of 74,338 firm‑quarter observations from 3,897 unique U.S. publicly listed firms between 2006 and 2020.

Recognised sources: CRSP Bloomberg Compustat

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • The predictive power of LLMs in the two forecasting tasks partly reflects memorization rather than genuine inference. predictive-accuracy
    Source sentence
    Applying the LAP test to two forecasting tasks—news headlines predicting stock returns and earnings‑call transcripts predicting capital expenditures—we find that the predictive power of LLMs partly reflects memorization rather than genuine inference.
  • The triple interaction term LLM × Small × LAP is positive and statistically significant, indicating lookahead bias drives stronger predictability for small‑cap stocks. statistical
    Source sentence
    The results confirm this suspect: the coefficient on the triple interaction term is positive and statistically significant.
  • LAP and model confidence capture distinct components of LLM forecasts. statistical
    Source sentence
    Taken together, the result suggests that LAP and model confidence capture distinct components of LLM forecasts.
  • In the out‑of‑sample placebo test, the interaction between LAP and LLM is insignificant, consistent with the absence of lookahead bias when the model lacks training‑period information. statistical
    Source sentence
    In the placebo regression, the interaction term between LAP and LLM is expected to yield an insignificant coefficient, consistent with the absence of lookahead bias once the model no longer has access to training‑period information.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 26 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 4 papers in the archive (4 by independent authors): Anonymization and Information Loss (uses its method or data); Look-Ahead-Bench: a Standardized Benchmark of Look-ahead … (uses its method or data)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (56)

  1. Quantifying Memorization Across Neural Language Models (2023)
  2. Scaling laws for fact memorization of large language models (2024) — link
  3. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  4. The ghost in the machine: Generating beliefs with large language models (2023) — link
  5. Dated data: Tracing knowledge cutoffs in large language models (2024)
  6. The llama 3 herd of models (2024) — link
  7. Attention is all you need (2017)
  8. LLaMA 3.3 70B Instruct Model Card (2024)
and 48 more
  1. Language models are few-shot learners (2020)
  2. Expected returns and large language models (2022)
  3. Generative AI for economic research: Use cases and implications for economists (2023)
  4. Experimental evidence on the productivity effects of generative artificial intelligence (2023)
  5. … and the cross-section of expected returns (2016)
  6. Leakage and the reproducibility crisis in ML-based science (2022) — link
  7. Extracting training data from large language models (2021)
  8. On the opportunities and risks of foundation models (2021) — link
  9. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975)
  10. How to talk when a machine is listening: Corporate disclosure in the age of AI (2023)
  11. Does finance benefit society? A language embedding approach (2025)
  12. Intellectual property protection lost and competition: An examination using large language models (2022)
  13. Can chatgpt decipher fedspeak? (2024)
  14. ChatGPT and Corporate Policies — → 2409.17933
  15. Investor sentiment and the cross-section of stock returns (2006)
  16. Harnessing Generative AI for Economic Insights — → 2410.03897
  17. Lookahead bias in pretrained language models (2024)
  18. Detecting pretraining data from large language models (2024)
  19. Entity neutering (2025)
  20. Chronologically Consistent Large Language Models — → 2502.21206
  21. Structured Beliefs and Fund Performance: An LLM -Based Approach (2024)
  22. Instruction Tuning Chronologically Consistent Language Models (2025) — link
  23. DatedGPT : Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining (2026) — link
  24. The Memorization Problem: Can We Trust LLMs’ Economic Forecasts? — → 2504.14765
  25. Why language models hallucinate (2025) — link
  26. Impact of pretraining term frequencies on few-shot reasoning (2022)
  27. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing (2023)
  28. StoriesLM: A Family of Language Models With Time-Indexed Training Data (2024)
  29. Out of the Black Box: Uncertainty Quantification for LLMs via Conditional Probabilities (2024)
  30. Memorization without overfitting: Analyzing the training dynamics of large language models (2022)
  31. Membership inference attacks from first principles (2022)
  32. Membership inference attacks against machine learning models (2017)
  33. On the Importance of Difficulty Calibration in Membership Inference Attacks (2022) — link
  34. Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023) — link
  35. Privacy risk in machine learning: Analyzing the connection to overfitting (2018)
  36. Anonymization and Information Loss — → 2511.15364
  37. An introduction to the bootstrap (1994)
  38. Assessing look-ahead bias in stock return predictions generated by gpt sentiment analysis — → 2309.17322
  39. Large language models: An applied econometric framework (2025)
  40. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  41. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
  42. Analysts’ Belief Formation in Their Own Words (2024)
  43. The Inefficient Pricing of News (2026)
  44. Scaling Point-in-Time Language Models (2026)
  45. Debiasing LLMs by Fine-tuning — → 2604.02921
  46. A Financial Brain Scan of the LLM — → 2508.21285
  47. Caution Ahead: Numerical Reasoning and Look-Ahead Bias in AI Models (2026) — link
  48. CFOs Meet LLMs (2026)

9 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.