Paper: arXiv 2504.14765
Authors: Alejandro Lopez-Lira, Yuehua Tang, Mingyin Zhu
Abstract
Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data. Counterfactual forecasting ability is non-identified when the model has seen the realized values: any observed output is consistent with both genuine skill and memorization. Any evidence of memorization represents only a lower bound on encoded knowledge. We demonstrate LLMs have memorized economic and financial data, recalling exact values before their knowledge cutoff. Instructions to respect historical boundaries fail to prevent recall-level accuracy, and masking fails as LLMs reconstruct entities and dates from minimal context. Post-cutoff, we observe no recall. Memorization extends to embeddings.
Complexity vs Empirical Score
- Math Complexity: 3.0/10
- Empirical Rigor: 7.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper’s theoretical component is primarily conceptual, focusing on non-identification and logical arguments, which is moderately complex; however, it is heavily grounded in systematic empirical testing with extensive data analysis, evaluation metrics, and specific findings on recall accuracy and masking failures.
Research Flowchart
flowchart TD Start["Research Goal: Can LLMs be trusted for economic forecasts?"] --> Method["Methodology: Test recall vs. genuine skill"] Method --> Data["Data: Historical economic/financial series"] Data --> Proc["Computational Process: Prompt recall & counterfactual tests"] Proc --> Out1["Finding 1: Models memorize exact values pre-cutoff"] Proc --> Out2["Finding 2: Instructions/masking fail to prevent recall"] Proc --> Out3["Finding 3: Post-cutoff forecasts show no recall"] Out1 --> Result["Outcome: Forecast trust is impossible pre-cutoff"] Out2 --> Result Out3 --> Result
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes that large language models cannot be trusted for economic forecasts within their training period because any observed output may be memorization rather than genuine skill.
Source sentence
Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data.
Assumes 3
- Deterministic (greedy) decoding is used for clarity in the analysis. model
Source sentence
We focus on deterministic (greedy) decoding for clarity.
- The training procedure is deterministic conditional on the data, so the only difference between full and restricted models is inclusion of future information. model
Source sentence
it assumes the training procedure is deterministic conditional on the data, such that the only difference between θ and θt is the inclusion of future information, I>t.
- Future‑invariant tasks produce the same optimal answer regardless of post‑t information. statistical
Source sentence
Future‑invariant: The task is future‑invariant if the identity of the optimal answer is invariant to post‑t information.
Data
- S&P 500, Dow Jones Industrial Average, Nasdaq Composite daily closing values market-prices public
Source sentence
We use the daily closing values of the S&P 500, the Dow Jones Industrial Average, and the Nasdaq Composite from Yahoo Finance to evaluate the LLM’s answers.
- Individual stock daily closing prices market-prices commercial
Source sentence
We use Center for Research in Security Prices (CRSP) data to obtain daily stock market data for individual stock closing prices.
- Macroeconomic indicators (GDP growth, inflation, unemployment, 10‑yr Treasury Yield, VIX, macro public
Source sentence
We obtain the actual unemployment rate and 10‑year Treasury Yield values from Federal Reserve Economic Data (FRED). We obtain the VIX levels from Yahoo Finance. For GDP growth, inflation, housing starts, and change in nonfarm payrolls, we use the Philadelphia Federal Reserve Real‑Time Data Set.
- WSJ front‑page headlines text-news commercial
Source sentence
The WSJ front‑page news dataset comprises 90,123 headlines from January 1990 to February 2025.
- Earnings conference call transcripts text-news commercial
Source sentence
The conference call dataset starts in January 2006 and ends in May 2024.
- Firm‑specific headlines text-news commercial
Source sentence
We arrive at a sample of 1,358,737 headlines covering 4,391 unique firms from January 2000 to June 2024.
RavenPack CRSP S&P Capital IQ Yahoo Finance FRED
Costs
Not applicable
Finds 5
- GPT‑4o perfectly recalls pre‑cutoff exact index levels with Mean Absolute Percent Errors below 2% and directional accuracy up to 80.7%, demonstrating extensive memorization. performance
Source sentence
For pre‑cutoff daily exact numerical levels, GPT‑4o exhibits strong recall, with Mean Absolute Percent Errors of 0.61% for S&P 500, 0.53% for DJIA, and 1.80% for Nasdaq Composite, and Directional Accuracy ranging from 69.4% (Nasdaq) to 80.7% (DJIA).
- Post‑cutoff performance collapses, with Mean Absolute Percent Errors rising to 13‑20% and directional accuracy dropping to near‑random (44‑49%). negative
Source sentence
In contrast, post‑cutoff performance collapses. For instance, without context, the Mean Absolute Percent Errors balloon to 13.01%–20.48% for exact levels and Directional Accuracy dropping to near‑random levels (44.0%–49.3%).
- GPT‑4o identifies headline dates with 98.5% year accuracy and 47.0% exact‑date accuracy for pre‑cutoff WSJ headlines, far above chance. predictive-accuracy
Source sentence
For pre‑cutoff headlines, it achieves 98.5% accuracy in determining the correct year and 90.4% in identifying the correct month and year. Even for exact date identification, the model achieves 47.0% accuracy—significantly above chance levels.
- Masking and prompting strategies fail: the model still reconstructs masked entities and dates, achieving high recall despite anonymization. theoretical
Source sentence
the model still correctly identified the company (ETH), quarter (Q1), and year (2018).
- Statistical tests with small post‑cutoff samples have low power, making memorization indistinguishable from genuine skill. statistical
Source sentence
When post‑cutoff sample size is small relative to pre‑cutoff sample size, genuine forecasting skill and undetected memorization cannot be reliably distinguished by hypothesis tests.
Checked by others
Supported by
- Thinking Fast, Trading Late: System One Decision Models vs the Market
Compare the findings
This paper: Masking and prompting strategies fail: the model still reconstructs masked entities and dates, achieving high recall despite anonymization.
That paper: Commercial models can still identify masked companies 62%–74% of the time against a 28% baseline, showing masking hides names but not firms.
18 papers in the archive : Seeing the Goal, Missing the Truth: Human Accountability for … ; Track or Follow? A Jev-Compatible Multi-Agent Trading System … ; Cross-Stock Predictability via LLM-Augmented Semantic … ; How Well Do LLMs Perform in Peer Review? Evidence from … ; Language Model Guided Reinforcement Learning in Quantitative …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.