Paper: arXiv 2512.23847
Authors: Zhenyu Gao, Wenxi Jiang, Yutong Yan
Abstract
We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs). Using state-of-the-art pre-training data detection techniques, we estimate the likelihood that a given prompt appeared in an LLM’s training corpus, a statistic we term Lookahead Propensity (LAP). We formally show that a positive correlation between LAP and forecast accuracy indicates the presence and magnitude of lookahead bias, and apply the test to two forecasting tasks: news headlines predicting stock returns and earnings call transcripts predicting capital expenditures. Our test provides a cost-efficient, diagnostic tool for assessing the validity and reliability of LLM-generated forecasts.
Complexity vs Empirical Score
- Math Complexity: 4.5/10
- Empirical Rigor: 7.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper develops a formal statistical test with econometric propositions but relies on established MIA metrics rather than novel, heavy derivations, placing it in the moderate math range. It demonstrates empirical rigor by applying the test to real financial datasets (news headlines and earnings calls) and validating with out-of-sample bootstrap analysis, though it lacks code or backtest-ready implementation details.
Research Flowchart
flowchart TD
A["Research Goal:<br>Test for Lookahead Bias<br>in LLM Economic Forecasts"] --> B{"Data & Input:<br>Financial Text Data"}
B --> C["Process: Estimate<br>Lookahead Propensity LAP<br>via P(data in training corpus)"]
C --> D{"Computation:<br>Correlation Analysis<br>LAP vs Forecast Accuracy"}
D --> E{"Statistical Test:<br>Significant Positive Correlation?"}
E -- Yes --> F["Outcome: Lookahead Bias<br>Detectable & Quantifiable"]
E -- No --> G["Outcome: Bias Absent<br>or Insufficient Data"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a statistical test based on Lookahead Propensity (LAP) to detect lookahead bias in LLM-generated economic forecasts.
Source sentence
We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs).
Assumes 4
- A positive correlation between LAP and forecast accuracy indicates lookahead bias rather than genuine predictive reasoning. statistical
Source sentence
The main proposition implies that if forecast accuracy is correlated with LAP, such a relationship indicates lookahead bias rather than genuine predictive reasoning.
- Headlines published before 4 p.m. are linked to the next‑day stock return; otherwise they are treated as news on day t + 1. execution
Source sentence
For simplicity, we use stock market close time 4 p.m. as the cutoff to classify headlines on day t.
- Llama‑3.3 provides access to prompt token probabilities needed to compute LAP. model
Source sentence
Most importantly, Llama-3.3 provides access to prompt token probabilities, which are essential for computing our LAP measure.
- Out‑of‑sample period starts at the model’s release date rather than its knowledge cutoff. other
Source sentence
To be conservative, we use the model’s release date—rather than its claimed knowledge cutoff—as the starting point for our out-of-sample
Data
- CRSP daily stock file market-prices commercial
Source sentence
on the New York Stock Exchange (NYSE), the National Association of Securities Dealers Automated Quotations (NASDAQ), and the American Stock Exchange (AMEX).
- Bloomberg News headlines text-news commercial
Source sentence
The final dataset consists of 91,361 news headlines covering 1,587 unique companies from January 2012 to December 2023.
- Thomson Reuters StreetEvents earnings call transcripts text-news commercial
Source sentence
our final sample consists of 74,338 firm‑quarter observations from 3,897 unique U.S. publicly listed firms between 2006 and 2020.
Costs
Not applicable
Finds 4
- The predictive power of LLMs in the two forecasting tasks partly reflects memorization rather than genuine inference. predictive-accuracy
Source sentence
Applying the LAP test to two forecasting tasks—news headlines predicting stock returns and earnings‑call transcripts predicting capital expenditures—we find that the predictive power of LLMs partly reflects memorization rather than genuine inference.
- The triple interaction term LLM × Small × LAP is positive and statistically significant, indicating lookahead bias drives stronger predictability for small‑cap stocks. statistical
Source sentence
The results confirm this suspect: the coefficient on the triple interaction term is positive and statistically significant.
- LAP and model confidence capture distinct components of LLM forecasts. statistical
Source sentence
Taken together, the result suggests that LAP and model confidence capture distinct components of LLM forecasts.
- In the out‑of‑sample placebo test, the interaction between LAP and LLM is insignificant, consistent with the absence of lookahead bias when the model lacks training‑period information. statistical
Source sentence
In the placebo regression, the interaction term between LAP and LLM is expected to yield an insignificant coefficient, consistent with the absence of lookahead bias once the model no longer has access to training‑period information.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 26 related papers). Coverage is partial, so this is neither replication nor refutation.
4 papers in the archive : Anonymization and Information Loss ; Look-Ahead-Bench: a Standardized Benchmark of Look-ahead …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (56)
- Quantifying Memorization Across Neural Language Models (2023)
- Scaling laws for fact memorization of large language models (2024) — link
- Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
- The ghost in the machine: Generating beliefs with large language models (2023) — link
- Dated data: Tracing knowledge cutoffs in large language models (2024)
- The llama 3 herd of models (2024) — link
- Attention is all you need (2017)
- LLaMA 3.3 70B Instruct Model Card (2024)
and 48 more
- Language models are few-shot learners (2020)
- Expected returns and large language models (2022)
- Generative AI for economic research: Use cases and implications for economists (2023)
- Experimental evidence on the productivity effects of generative artificial intelligence (2023)
- … and the cross-section of expected returns (2016)
- Leakage and the reproducibility crisis in ML-based science (2022) — link
- Extracting training data from large language models (2021)
- On the opportunities and risks of foundation models (2021) — link
- Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975)
- How to talk when a machine is listening: Corporate disclosure in the age of AI (2023)
- Does finance benefit society? A language embedding approach (2025)
- Intellectual property protection lost and competition: An examination using large language models (2022)
- Can chatgpt decipher fedspeak? (2024)
- ChatGPT and Corporate Policies — → 2409.17933
- Investor sentiment and the cross-section of stock returns (2006)
- Harnessing Generative AI for Economic Insights — → 2410.03897
- Lookahead bias in pretrained language models (2024)
- Detecting pretraining data from large language models (2024)
- Entity neutering (2025)
- Chronologically Consistent Large Language Models — → 2502.21206
- Structured Beliefs and Fund Performance: An LLM -Based Approach (2024)
- Instruction Tuning Chronologically Consistent Language Models (2025) — link
- DatedGPT : Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining (2026) — link
- The Memorization Problem: Can We Trust LLMs’ Economic Forecasts? — → 2504.14765
- Why language models hallucinate (2025) — link
- Impact of pretraining term frequencies on few-shot reasoning (2022)
- Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing (2023)
- StoriesLM: A Family of Language Models With Time-Indexed Training Data (2024)
- Out of the Black Box: Uncertainty Quantification for LLMs via Conditional Probabilities (2024)
- Memorization without overfitting: Analyzing the training dynamics of large language models (2022)
- Membership inference attacks from first principles (2022)
- Membership inference attacks against machine learning models (2017)
- On the Importance of Difficulty Calibration in Membership Inference Attacks (2022) — link
- Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023) — link
- Privacy risk in machine learning: Analyzing the connection to overfitting (2018)
- Anonymization and Information Loss — → 2511.15364
- An introduction to the bootstrap (1994)
- Assessing look-ahead bias in stock return predictions generated by gpt sentiment analysis — → 2309.17322
- Large language models: An applied econometric framework (2025)
- Llama 2: Open foundation and fine-tuned chat models (2023) — link
- Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
- Analysts’ Belief Formation in Their Own Words (2024)
- The Inefficient Pricing of News (2026)
- Scaling Point-in-Time Language Models (2026)
- Debiasing LLMs by Fine-tuning — → 2604.02921
- A Financial Brain Scan of the LLM — → 2508.21285
- Caution Ahead: Numerical Reasoning and Look-Ahead Bias in AI Models (2026) — link
- CFOs Meet LLMs (2026)
9 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.