Paper: arXiv 2502.21206

Authors: Songrun He, Linying Lv, Asaf Manela, Jimmy Wu

Abstract

Large language models are increasingly used in social sciences, but their training data can introduce lookahead bias and training leakage. A good chronologically consistent language model requires efficient use of training data to maintain accuracy despite time-restricted data. Here, we overcome this challenge by training a suite of chronologically consistent large language models, ChronoBERT and ChronoGPT, which incorporate only the text data that would have been available at each point in time. Despite this strict temporal constraint, our models achieve strong performance on natural language processing benchmarks, outperforming or matching widely used models (e.g., BERT), and remain competitive with larger open-weight models. Lookahead bias is model and application-specific because even if a chronologically consistent language model has poorer language comprehension, a regression or prediction model applied on top of the language model can compensate. In an asset pricing application predicting next-day stock returns from financial news, we find that ChronoBERT and ChronoGPT’s real-time outputs achieve Sharpe ratios comparable to a much larger Llama model, indicating that lookahead bias is modest. Our results demonstrate a scalable, practical framework to mitigate training leakage, ensuring more credible backtests and predictions across finance and other social science domains.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; reported Sharpe ratio of 4.92 is implausibly high for daily or lower-frequency data; see penalty flags)
  • Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 7.0)

Why this score: The paper employs sophisticated deep learning architectures (BERT, GPT) and complex training regimes, but the math is largely applied from established NLP literature rather than introducing novel theory. Empirical rigor is high, featuring extensive backtesting on financial news data, Sharpe ratio analysis, and comparisons to baselines, demonstrating a data-heavy, implementation-focused study.

Research Flowchart

  flowchart TD
  A["Research Goal: <br>Mitigate Lookahead Bias in LLMs <br>for Social Science"] --> B["Methodology: <br>Chronological Training"]
  B --> C{"Data Inputs: <br>Time-Restrained Text Data"}
  C --> D["Computational Process: <br>Train ChronoBERT & ChronoGPT"]
  D --> E["Outcomes 1: <br>NLP Benchmark Performance <br>(Match/Met BERT)"]
  D --> F["Outcomes 2: <br>Asset Pricing Application <br>(Stock Returns Prediction)"]
  F --> G["Result: <br>Comparable Sharpe Ratios <br>(Modest Lookahead Bias)"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper introduces ChronoBERT and ChronoGPT, chronologically consistent large language models trained only on text available at each point in time.

Source sentence
we overcome this challenge by training a suite of chronologically consistent large language models, ChronoBERT and ChronoGPT, which incorporate only the text data that would have been available at each point in time.

Assumes 1

  • Market efficiency tests assume that prices reflect only known facts at the time. market
    Source sentence
    Market efficiency tests (Fama, 1970) assume that prices reflect only known facts at the time.

Data

Backtest sample: 2008-01 – 2023-07 · US stocks · equities · daily

  • Dow Jones Newswire dataset · Dow Jones (subscribed by major institutional investors) text-news commercial
    firm-specific news headlines within trading day window
    Source sentence
    ... For each firm-day observation, we aggregate all news headlines related to the firm within the trading day window—spanning from 4:00 p.m. EST on day t −1 to 4:00 p.m. EST on day t—and treat the combined text as the firm’s textual information.

Costs

Not modeled A trading or execution result is reported without costs.

Finds 4

Reported performance (ChronoGPTRealtime): Sharpe 4.92

  • ChronoBERTRealtime and ChronoGPTRealtime generate Sharpe ratios of 4.80 and 4.92, outperforming GPT-2 XL, ModernBERT, BERT, FinBERT and StoriesLM. performance
    Source sentence
    In this news return prediction setting, the H-L portfolios from ChronoBERTRealtime and ChronoGPTRealtime generate Sharpe ratios of 4.80 and 4.92, outperforming GPT-2 XL, ModernBERT, BERT, FinBERT and StoriesLM.
  • ChronoBERT and ChronoGPT have comparable Sharpe ratios to Llama 3.1 with no statistically significant differences (p‑value 0.315 and 0.547). performance
    Source sentence
    Comparing the investment performance of ChronoBERT and ChronoGPT against the Llama 3.1 model, we find that ChronoBERT and ChronoGPT generate comparable Sharpe ratios, with no statistically significant differences (p-value of 0.315 and 0.547).
  • Lookahead bias appears modest in the stock return forecasting application. statistical
    Source sentence
    A somewhat surprising finding is that lookahead bias appears to be modest in this stock return forecasting application.
  • ChronoBERT and ChronoGPT match or surpass the language comprehension abilities of BERT and GPT-2 while achieving performance comparable to the much larger Llama 3.1 model. predictive-accuracy
    Source sentence
    Our results show that ChronoBERT and ChronoGPT match or surpass the language comprehension abilities of BERT and GPT-2 while generating performance comparable to the much larger Llama 3.1 model in asset pricing applications.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 41 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 13 papers in the archive (12 by independent authors): Re(Visiting) Time Series Foundation Models in Finance (discusses its findings); A Test of Lookahead Bias in LLM Forecasts (uses its method or data); Continuous Risk Factor Models: Analyzing Asset Correlations … (uses its method or data)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (85)

  1. Lookahead Bias in Pretrained Language Models (2024) — link
  2. Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis — → 2309.17322
  3. Documenting Large Webtext Corpora : A Case Study on the Colossal Clean Crawled Corpus (2021) — link
  4. StoriesLM : A Family of Language Models With Time-Indexed Training Data (2024)
  5. American Stories : A Large - Scale Structured Text Dataset of Historical U . S . Newspapers (2023) — link
  6. Improving Language Understanding by Generative Pre - Training (2018)
  7. RoBERTa : A Robustly Optimized BERT Pretraining Approach (2019) — link
  8. Language Models are Few - Shot Learners (2020) — link
and 77 more
  1. Text as Data (2019) — link
  2. Scaling laws for neural language models (2020) — link
  3. 2 OLMo 2 Furious (2024) — link
  4. Prediction Policy Problems (2015) — link
  5. The Persistence of Risk - Adjusted Mutual Fund Performance (1996) — link
  6. Leakage and the Reproducibility Crisis in ML -based Science (2022) — link
  7. The secret sharer: Evaluating and testing unintended memorization in neural networks (2019)
  8. Quantifying memorization across neural language models (2022) — link
  9. Documenting Large Webtext Corpora : A Case Study on the Colossal Clean Crawled Corpus (2021) — link
  10. Time Travel in LLMs : Tracing Data Contamination in Large Language Models (2024) — link
  11. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023) — link
  12. Training Data Leakage Analysis in Language Models (2021) — link
  13. Proving Test Set Contamination in Black Box Language Models (2023) — link
  14. Surveying Generative AI ’s Economic Expectations (2023) — link
  15. AI recognition of patient race in medical imaging: a modelling study (2022) — link
  16. Does Finance Benefit Society ? A Language Embedding Approach (2022) — link
  17. BloombergGPT : A Large Language Model for Finance (2023) — link
  18. In- Context Unlearning : Language Models as Few Shot Unlearners (2024) — link
  19. Time is Encoded in the Weights of Finetuned Language Models (2023) — link
  20. The promise and peril of generative AI : Evidence from ChatGPT as sell-side analysts (2023)
  21. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
  22. Approaching Human - Level Forecasting with Language Models (2024) — link
  23. hmBERT : Historical Multilingual Language Models for Named Entity Recognition (2022) — link
  24. MacBERTh : Development and Evaluation of a Historically Pre -trained Language Model for English (1450-1950) (2021) — link
  25. Synthetic dataset generation techniques: generating custom sentence similarity data (2024) — link
  26. Yahoo! for Amazon : Sentiment Extraction from Small Talk on the Web (2007) — link
  27. Giving Content to Investor Sentiment: The Role of Media in the Stock Market (2007) — link
  28. More Than Words: Quantifying Language to Measure Firms’ Fundamentals (2008) — link
  29. How news and its context drive risk and returns around the world (2019) — link
  30. Predicting Returns With Text Data (2019)
  31. The colour of finance words (2023) — link
  32. Time Variation in the News – Returns Relationship (2023) — link
  33. Can chatgpt decipher fedspeak (2023) — link
  34. Memorization Without Overfitting : Analyzing the Training Dynamics of Large Language Models (2022) — link
  35. Extracting training data from large language models (2021)
  36. Upstream Mitigation Is \textit N ot All You Need : Testing the Bias Transfer Hypothesis in Pre - Trained Language Models (2022) — link
  37. Expected Returns and Large Language Models (2023) — link
  38. Distributed representations of words and phrases and their compositionality (2013)
  39. Glove: Global vectors for word representation (2014)
  40. Attention is all you need (2017)
  41. Language models are unsupervised multitask learners (2019)
  42. How to Learn and Teach Economics with Large Language Models , Including GPT (2023) — link
  43. Experimental evidence on the productivity effects of generative artificial intelligence (2023) — link
  44. Hellaswag: Can a machine really finish your sentence? (2019)
  45. ChatGPT, Stock Market Predictability and Links to the Macroeconomy (2023)
  46. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference (2024) — link
  47. MosaicBERT : A bidirectional encoder optimized for fast pretraining (2023)
  48. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  49. Textbooks are all you need (2023) — link
  50. The fineweb datasets: Decanting the web for the finest text data at scale (2024) — link
  51. Scaling data-constrained language models (2023)
  52. GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2019) — link
  53. FinBERT : A large language model for extracting information from financial text (2023)
  54. Robust performance hypothesis testing with the Sharpe ratio (2008)
  55. The Natural Language of Finance (2025)
  56. modded-nanogpt: Speedrunning the NanoGPT baseline (2024) — link
  57. Build NanoGPT (2024) — link
  58. Training compute-optimal large language models (2022) — link
  59. Pervasive underreaction: Evidence from high-frequency data (2021)
  60. Empirical Asset Pricing with Probability Forecasts (2024)
  61. Large language models: An applied econometric framework (2025)
  62. The Value of Information from Sell-side Analysts — → 2411.13813
  63. News implied volatility and disaster concerns (2017)
  64. Text selection (2021)
  65. Does finance benefit society? A language embedding approach (2025)
  66. Business news and business cycles (2024)
  67. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks (2011)
  68. The llama 3 herd of models (2024) — link
  69. Efficient capital markets (1970)
  70. Why is it okay to average embeddings? (2020)
  71. HellaSwag: Can a Machine Really Finish Your Sentence? (2019) — link
  72. Re(Visiting) Large Language Models in Finance (2024)
  73. Caution Ahead : Numerical Reasoning and Look-ahead Bias in AI Models (2024) — link
  74. Crash narratives (2022)
  75. Entity Neutering (2025)
  76. The Memorization Problem : Can We Trust LLMs ’ Economic Forecasts ? — → 2504.14765
  77. AI ’s Predictable Memory and Its Implications for Finance (2025) — link

4 of 85 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.