{"paper_id":"2504.14765","title":"The Memorization Problem: Can We Trust LLMs' Economic Forecasts?","authors":["Alejandro Lopez-Lira","Yuehua Tang","Mingyin Zhu"],"paper_date":"2025-04-20","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2504.14765v2","page_url":"https://thequant.space/flowcharts/the-memorization-problem-can-we-trust-llms-economic-foreca/","scores":{"empirical_rigor":7.5,"math_complexity":3,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes that large language models cannot be trusted for economic forecasts within their training period because any observed output may be memorization rather than genuine skill.","quote":"Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data.","summary":"Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data. Counterfactual forecasting ability is non-identified when the model has seen the realized values: any observed output is consistent with both genuine skill and memorization. Any…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Deterministic (greedy) decoding is used for clarity in the analysis.","type":"model","stated":"explicit","quote":"We focus on deterministic (greedy) decoding for clarity."},{"assumption":"The training procedure is deterministic conditional on the data, so the only difference between full and restricted models is inclusion of future information.","type":"model","stated":"explicit","quote":"it assumes the training procedure is deterministic conditional on the data, such that the only difference between θ and θt is the inclusion of future information, I>t."},{"assumption":"Future‑invariant tasks produce the same optimal answer regardless of post‑t information.","type":"statistical","stated":"explicit","quote":"Future‑invariant: The task is future‑invariant if the identity of the optimal answer is invariant to post‑t information."}]},"datasets":{"status":"available","items":[{"name":"S&P 500, Dow Jones Industrial Average, Nasdaq Composite daily closing values","provider":"Yahoo Finance","kind":"market-prices","access":"public","period":"January 1990 to September 2023","universe":null,"quote":"We use the daily closing values of the S&P 500, the Dow Jones Industrial Average, and the Nasdaq Composite from Yahoo Finance to evaluate the LLM’s answers.","known_id":"yahoo-finance"},{"name":"Individual stock daily closing prices","provider":"Center for Research in Security Prices (CRSP)","kind":"market-prices","access":"commercial","period":"January 1990 to September 2023","universe":"Magnificent 7 and randomly selected subsamples totaling over 4,200 unique ticker symbols","quote":"We use Center for Research in Security Prices (CRSP) data to obtain daily stock market data for individual stock closing prices.","known_id":"crsp"},{"name":"Macroeconomic indicators (GDP growth, inflation, unemployment, 10‑yr Treasury Yield, VIX, ","provider":"FRED, Yahoo Finance, Philadelphia Federal Reserve Real‑Time ","kind":"macro","access":"public","period":"January 1990 to September 2023","universe":null,"quote":"We obtain the actual unemployment rate and 10‑year Treasury Yield values from Federal Reserve Economic Data (FRED). We obtain the VIX levels from Yahoo Finance. For GDP growth, inflation, housing starts, and change in nonfarm payrolls, we use the Philadelphia Federal Reserve Real‑Time Data Set.","known_id":"fred"},{"name":"WSJ front‑page headlines","provider":"Factiva","kind":"text-news","access":"commercial","period":"January 1990 to February 2025","universe":"90,123 headlines (~9 per date)","quote":"The WSJ front‑page news dataset comprises 90,123 headlines from January 1990 to February 2025."},{"name":"Earnings conference call transcripts","provider":"Capital IQ","kind":"text-news","access":"commercial","period":"January 2006 to May 2024","universe":null,"quote":"The conference call dataset starts in January 2006 and ends in May 2024.","known_id":"capital-iq"},{"name":"Firm‑specific headlines","provider":"RavenPack news database","kind":"text-news","access":"commercial","period":"January 2000 to June 2024","universe":"1,358,737 headlines covering 4,391 unique firms","quote":"We arrive at a sample of 1,358,737 headlines covering 4,391 unique firms from January 2000 to June 2024.","known_id":"ravenpack"}],"known":[{"id":"ravenpack","name":"RavenPack","provider":"RavenPack","kind":"text-news","access":"commercial","quote":"The textual data we use include The Wall Street Journal (WSJ) front-page headlines obtained from Factiva, earnings conference call transcripts from Capital IQ, firm-specific headlines, and the RavenPack news database.","mentions":6,"used_in":4},{"id":"crsp","name":"CRSP","provider":"Center for Research in Security Prices","kind":"market-prices","access":"commercial","quote":"We randomly sample 50 stocks from each quintile and resampled annually to account for size changes, using stock price data from CRSP.","mentions":4,"used_in":3},{"id":"capital-iq","name":"S&P Capital IQ","provider":"S&P Global","kind":"fundamentals","access":"commercial","quote":"The textual data we use include The Wall Street Journal (WSJ) front-page headlines obtained from Factiva, earnings conference call transcripts from Capital IQ, firm-specific headlines, and the RavenPack news database.","mentions":2,"used_in":2},{"id":"yahoo-finance","name":"Yahoo Finance","provider":"Yahoo","kind":"market-prices","access":"public","quote":"We use the daily closing values of the S&P 500, the Dow Jones Industrial Average, and the Nasdaq Composite from Yahoo Finance to evaluate the LLM’s answers.","mentions":2,"used_in":2},{"id":"fred","name":"FRED","provider":"Federal Reserve Bank of St. Louis","kind":"macro","access":"public","quote":"We obtain the actual unemploy-ment rate and 10-year Treasury Yield values from Federal Reserve Economic Data (FRED).","mentions":2,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"GPT‑4o perfectly recalls pre‑cutoff exact index levels with Mean Absolute Percent Errors below 2% and directional accuracy up to 80.7%, demonstrating extensive memorization.","kind":"performance","quote":"For pre‑cutoff daily exact numerical levels, GPT‑4o exhibits strong recall, with Mean Absolute Percent Errors of 0.61% for S&P 500, 0.53% for DJIA, and 1.80% for Nasdaq Composite, and Directional Accuracy ranging from 69.4% (Nasdaq) to 80.7% (DJIA)."},{"finding":"Post‑cutoff performance collapses, with Mean Absolute Percent Errors rising to 13‑20% and directional accuracy dropping to near‑random (44‑49%).","kind":"negative","quote":"In contrast, post‑cutoff performance collapses. For instance, without context, the Mean Absolute Percent Errors balloon to 13.01%–20.48% for exact levels and Directional Accuracy dropping to near‑random levels (44.0%–49.3%)."},{"finding":"GPT‑4o identifies headline dates with 98.5% year accuracy and 47.0% exact‑date accuracy for pre‑cutoff WSJ headlines, far above chance.","kind":"predictive-accuracy","quote":"For pre‑cutoff headlines, it achieves 98.5% accuracy in determining the correct year and 90.4% in identifying the correct month and year. Even for exact date identification, the model achieves 47.0% accuracy—significantly above chance levels."},{"finding":"Masking and prompting strategies fail: the model still reconstructs masked entities and dates, achieving high recall despite anonymization.","kind":"theoretical","quote":"the model still correctly identified the company (ETH), quarter (Q1), and year (2018)."},{"finding":"Statistical tests with small post‑cutoff samples have low power, making memorization indistinguishable from genuine skill.","kind":"statistical","quote":"When post‑cutoff sample size is small relative to pre‑cutoff sample size, genuine forecasting skill and undetected memorization cannot be reliably distinguished by hypothesis tests."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[{"paper":{"paper_id":"2611.03985","title":"Thinking Fast, Trading Late: System One Decision Models vs the Market","paper_date":"2026-11-02","authors":["Amir Dehkordi","Alireza Kargarzadeh","Nariman Khaledian","Navid Parvini","Sid Ghatak","Arman Khaledian"],"empirical_rigor":9,"page_url":"https://thequant.space/flowcharts/thinking-fast-trading-late-system-one-decision-models-vs-the-market/"},"question":"Does masking effectively hide entity identities in LLM economic forecasts and decision models?","this_paper_finding":"Masking and prompting strategies fail: the model still reconstructs masked entities and dates, achieving high recall despite anonymization.","other_paper_finding":"Commercial models can still identify masked companies 62%–74% of the time against a 28% baseline, showing masking hides names but not firms.","other_paper_quote":"Commercial models identify masked companies in 62% to 74% of cases against a 28% baseline: masking hides names, not companies.","note":"Both findings show masking fails: A retains high recall of masked entities/dates; B identifies masked companies 62‑74% vs 28% baseline.","confidence":0.776}],"contradicts":[],"qualifies":[],"compared_with":25,"citations":{"cited_by_in_corpus":18,"by_independent_authors":17,"how_cited":{"used_as_baseline":0,"uses_method_or_data":3,"extends":1,"discusses_findings":2,"mentions":10,"unclassified":2},"independent_papers":[{"paper":{"paper_id":"2602.09504","title":"Seeing the Goal, Missing the Truth: Human Accountability for AI Bias","paper_date":"2026-02-10","authors":["Sean Cao","Wei Jiang","Hui Xu"],"empirical_rigor":8,"page_url":"https://thequant.space/flowcharts/seeing-the-goal-missing-the-truth-human-accountability-for-ai-bias/"},"how":"discusses_findings","quote":"for instance, document that language models internalize economic and financial data prior to their knowledge cutoff dates, and that prompt-based instructions explicitly forbidding the use of such knowledge are insufficient to prevent its recall."},{"paper":{"paper_id":"2611.10414","title":"Track or Follow? A Jev-Compatible Multi-Agent Trading System When Web News Lags the Market","paper_date":"2026-11-07","authors":["Yebo Feng","Guangyan Gan","Liangrui Pan","Ruichao Liang","Cong Wu","Jiahua Xu"],"empirical_rigor":8,"page_url":"https://thequant.space/flowcharts/track-or-follow-a-jev-compatible-multi-agent-trading-system-when-web-news-lags/"},"how":"discusses_findings","quote":"prompts to ignore later in-formation do not help [33]."},{"paper":{"paper_id":"2604.19476","title":"Cross-Stock Predictability via LLM-Augmented Semantic Networks","paper_date":"2026-04-21","authors":["Yikuan Huang","Zheqi Fan","Kaiqi Hu","Yifan Ye"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/cross-stock-predictability-via-llm-augmented-semantic-networks/"},"how":"extends","quote":null},{"paper":{"paper_id":"2610.18435","title":"How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting","paper_date":"2026-10-13","authors":["Hai Lu","Yichen Luo","Luping Yu"],"empirical_rigor":8,"page_url":"https://thequant.space/flowcharts/how-well-do-llms-perform-in-peer-review-evidence-from-finance-and-accounting/"},"how":"uses_method_or_data","quote":null},{"paper":{"paper_id":"2508.02366","title":"Language Model Guided Reinforcement Learning in Quantitative Trading","paper_date":"2025-08-04","authors":["Adam Darmanin","Vince Vella"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/language-model-guided-reinforcement-learning-in-quantitative/"},"how":"uses_method_or_data","quote":null},{"paper":{"paper_id":"2603.23300","title":"Designing Agentic AI-Based Screening for Portfolio Investment","paper_date":"2026-03-24","authors":["Mehmet Caner","Agostino Capponi","Nathan Sun","Jonathan Y. Tan"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/designing-agentic-ai-based-screening-for-portfolio-investment/"},"how":"uses_method_or_data","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:40:49Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}