{"paper_id":"2512.23847","title":"A Test of Lookahead Bias in LLM Forecasts","authors":["Zhenyu Gao","Wenxi Jiang","Yutong Yan"],"paper_date":"2025-12-29","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2512.23847v1","page_url":"https://thequant.space/flowcharts/a-test-of-lookahead-bias-in-llm-forecasts/","scores":{"empirical_rigor":7,"math_complexity":4.5,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes a statistical test based on Lookahead Propensity (LAP) to detect lookahead bias in LLM-generated economic forecasts.","quote":"We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs).","summary":"We develop a statistical test to detect lookahead bias in economic forecasts generated by large language models (LLMs). Using state-of-the-art pre-training data detection techniques, we estimate the likelihood that a given prompt appeared in an LLM’s training corpus, a statistic we term Lookahead…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"A positive correlation between LAP and forecast accuracy indicates lookahead bias rather than genuine predictive reasoning.","type":"statistical","stated":"implied","quote":"The main proposition implies that if forecast accuracy is correlated with LAP, such a relationship indicates lookahead bias rather than genuine predictive reasoning."},{"assumption":"Headlines published before 4 p.m. are linked to the next‑day stock return; otherwise they are treated as news on day t + 1.","type":"execution","stated":"explicit","quote":"For simplicity, we use stock market close time 4 p.m. as the cutoff to classify headlines on day t."},{"assumption":"Llama‑3.3 provides access to prompt token probabilities needed to compute LAP.","type":"model","stated":"explicit","quote":"Most importantly, Llama-3.3 provides access to prompt token probabilities, which are essential for computing our LAP measure."},{"assumption":"Out‑of‑sample period starts at the model’s release date rather than its knowledge cutoff.","type":"other","stated":"explicit","quote":"To be conservative, we use the model’s release date—rather than its claimed knowledge cutoff—as the starting point for our out-of-sample"}]},"datasets":{"status":"available","items":[{"name":"CRSP daily stock file","provider":"CRSP","kind":"market-prices","access":"commercial","period":"all common stocks listed on NYSE, NASDAQ","universe":"common stocks with share code 10 or 11","quote":"on the New York Stock Exchange (NYSE), the National Association of Securities Dealers Automated Quotations (NASDAQ), and the American Stock Exchange (AMEX).","known_id":"crsp"},{"name":"Bloomberg News headlines","provider":"Bloomberg L.P.","kind":"text-news","access":"commercial","period":"January 2012 to December 2023","universe":"91,361 news headlines for 1,587 unique companies","quote":"The final dataset consists of 91,361 news headlines covering 1,587 unique companies from January 2012 to December 2023.","known_id":"bloomberg"},{"name":"Thomson Reuters StreetEvents earnings call transcripts","provider":"Thomson Reuters","kind":"text-news","access":"commercial","period":"2006 to 2020","universe":"74,338 firm‑quarter observations from 3,897 unique U.S. publicly listed firms","quote":"our final sample consists of 74,338 firm‑quarter observations from 3,897 unique U.S. publicly listed firms between 2006 and 2020.","known_id":"refinitiv"}],"known":[{"id":"crsp","name":"CRSP","provider":"Center for Research in Security Prices","kind":"market-prices","access":"commercial","quote":"outlets. We collect the news headlines for all CRSP-listed companies in the sam-ple period and match them based on company names and ticker symbols.","mentions":4,"used_in":3},{"id":"bloomberg","name":"Bloomberg","provider":"Bloomberg L.P.","kind":"market-prices","access":"commercial","quote":"First, we use news headlines from Bloomberg, whereas 5We also conduct robustness tests by replacing “stock price” with “stock return”, as returns more directly re-flect changes in market expectations.","mentions":4,"used_in":2},{"id":"compustat","name":"Compustat","provider":"S&P Global","kind":"fundamentals","access":"commercial","quote":"We obtain financial statement variables from Compustat.","mentions":2,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"The predictive power of LLMs in the two forecasting tasks partly reflects memorization rather than genuine inference.","kind":"predictive-accuracy","quote":"Applying the LAP test to two forecasting tasks—news headlines predicting stock returns and earnings‑call transcripts predicting capital expenditures—we find that the predictive power of LLMs partly reflects memorization rather than genuine inference."},{"finding":"The triple interaction term LLM × Small × LAP is positive and statistically significant, indicating lookahead bias drives stronger predictability for small‑cap stocks.","kind":"statistical","quote":"The results confirm this suspect: the coefficient on the triple interaction term is positive and statistically significant."},{"finding":"LAP and model confidence capture distinct components of LLM forecasts.","kind":"statistical","quote":"Taken together, the result suggests that LAP and model confidence capture distinct components of LLM forecasts."},{"finding":"In the out‑of‑sample placebo test, the interaction between LAP and LLM is insignificant, consistent with the absence of lookahead bias when the model lacks training‑period information.","kind":"statistical","quote":"In the placebo regression, the interaction term between LAP and LLM is expected to yield an insignificant coefficient, consistent with the absence of lookahead bias once the model no longer has access to training‑period information."}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":26,"citations":{"cited_by_in_corpus":4,"by_independent_authors":4,"how_cited":{"used_as_baseline":0,"uses_method_or_data":2,"extends":0,"discusses_findings":0,"mentions":2,"unclassified":0},"independent_papers":[{"paper":{"paper_id":"2511.15364","title":"Anonymization and Information Loss","paper_date":"2025-11-19","authors":["Ke Wu","Baozhong Yang","Zhenkun Ying","Dexin Zhou"],"empirical_rigor":8.5,"page_url":"https://thequant.space/flowcharts/anonymization-and-information-loss/"},"how":"uses_method_or_data","quote":null},{"paper":{"paper_id":"2601.13770","title":"Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance","paper_date":"2026-01-20","authors":["Mostapha Benhenda"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/look-ahead-bench-a-standardized-benchmark-of-look-ahead-bia/"},"how":"uses_method_or_data","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:17:49Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}