Paper: arXiv 2502.21206
Authors: Songrun He, Linying Lv, Asaf Manela, Jimmy Wu
Abstract
Large language models are increasingly used in social sciences, but their training data can introduce lookahead bias and training leakage. A good chronologically consistent language model requires efficient use of training data to maintain accuracy despite time-restricted data. Here, we overcome this challenge by training a suite of chronologically consistent large language models, ChronoBERT and ChronoGPT, which incorporate only the text data that would have been available at each point in time. Despite this strict temporal constraint, our models achieve strong performance on natural language processing benchmarks, outperforming or matching widely used models (e.g., BERT), and remain competitive with larger open-weight models. Lookahead bias is model and application-specific because even if a chronologically consistent language model has poorer language comprehension, a regression or prediction model applied on top of the language model can compensate. In an asset pricing application predicting next-day stock returns from financial news, we find that ChronoBERT and ChronoGPT’s real-time outputs achieve Sharpe ratios comparable to a much larger Llama model, indicating that lookahead bias is modest. Our results demonstrate a scalable, practical framework to mitigate training leakage, ensuring more credible backtests and predictions across finance and other social science domains.
Complexity vs Empirical Score
- Math Complexity: 3.0/10
- Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; reported Sharpe ratio of 4.92 is implausibly high for daily or lower-frequency data; see penalty flags)
- Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 7.0)
Why this score: The paper employs sophisticated deep learning architectures (BERT, GPT) and complex training regimes, but the math is largely applied from established NLP literature rather than introducing novel theory. Empirical rigor is high, featuring extensive backtesting on financial news data, Sharpe ratio analysis, and comparisons to baselines, demonstrating a data-heavy, implementation-focused study.
Research Flowchart
flowchart TD
A["Research Goal: <br>Mitigate Lookahead Bias in LLMs <br>for Social Science"] --> B["Methodology: <br>Chronological Training"]
B --> C{"Data Inputs: <br>Time-Restrained Text Data"}
C --> D["Computational Process: <br>Train ChronoBERT & ChronoGPT"]
D --> E["Outcomes 1: <br>NLP Benchmark Performance <br>(Match/Met BERT)"]
D --> F["Outcomes 2: <br>Asset Pricing Application <br>(Stock Returns Prediction)"]
F --> G["Result: <br>Comparable Sharpe Ratios <br>(Modest Lookahead Bias)"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper introduces ChronoBERT and ChronoGPT, chronologically consistent large language models trained only on text available at each point in time.
Source sentence
we overcome this challenge by training a suite of chronologically consistent large language models, ChronoBERT and ChronoGPT, which incorporate only the text data that would have been available at each point in time.
Assumes 1
- Market efficiency tests assume that prices reflect only known facts at the time. market
Source sentence
Market efficiency tests (Fama, 1970) assume that prices reflect only known facts at the time.
Data
2008-01 – 2023-07 · US stocks · equities · daily
- Dow Jones Newswire dataset text-news commercial
Source sentence
... For each firm-day observation, we aggregate all news headlines related to the firm within the trading day window—spanning from 4:00 p.m. EST on day t −1 to 4:00 p.m. EST on day t—and treat the combined text as the firm’s textual information.
Costs
Not modeled
Finds 4
Sharpe 4.92
- ChronoBERTRealtime and ChronoGPTRealtime generate Sharpe ratios of 4.80 and 4.92, outperforming GPT-2 XL, ModernBERT, BERT, FinBERT and StoriesLM. performance
Source sentence
In this news return prediction setting, the H-L portfolios from ChronoBERTRealtime and ChronoGPTRealtime generate Sharpe ratios of 4.80 and 4.92, outperforming GPT-2 XL, ModernBERT, BERT, FinBERT and StoriesLM.
- ChronoBERT and ChronoGPT have comparable Sharpe ratios to Llama 3.1 with no statistically significant differences (p‑value 0.315 and 0.547). performance
Source sentence
Comparing the investment performance of ChronoBERT and ChronoGPT against the Llama 3.1 model, we find that ChronoBERT and ChronoGPT generate comparable Sharpe ratios, with no statistically significant differences (p-value of 0.315 and 0.547).
- Lookahead bias appears modest in the stock return forecasting application. statistical
Source sentence
A somewhat surprising finding is that lookahead bias appears to be modest in this stock return forecasting application.
- ChronoBERT and ChronoGPT match or surpass the language comprehension abilities of BERT and GPT-2 while achieving performance comparable to the much larger Llama 3.1 model. predictive-accuracy
Source sentence
Our results show that ChronoBERT and ChronoGPT match or surpass the language comprehension abilities of BERT and GPT-2 while generating performance comparable to the much larger Llama 3.1 model in asset pricing applications.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 41 related papers). Coverage is partial, so this is neither replication nor refutation.
13 papers in the archive : Re(Visiting) Time Series Foundation Models in Finance ; A Test of Lookahead Bias in LLM Forecasts ; Continuous Risk Factor Models: Analyzing Asset Correlations …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (85)
- Lookahead Bias in Pretrained Language Models (2024) — link
- Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis — → 2309.17322
- Documenting Large Webtext Corpora : A Case Study on the Colossal Clean Crawled Corpus (2021) — link
- StoriesLM : A Family of Language Models With Time-Indexed Training Data (2024)
- American Stories : A Large - Scale Structured Text Dataset of Historical U . S . Newspapers (2023) — link
- Improving Language Understanding by Generative Pre - Training (2018)
- RoBERTa : A Robustly Optimized BERT Pretraining Approach (2019) — link
- Language Models are Few - Shot Learners (2020) — link
and 77 more
- Text as Data (2019) — link
- Scaling laws for neural language models (2020) — link
- 2 OLMo 2 Furious (2024) — link
- Prediction Policy Problems (2015) — link
- The Persistence of Risk - Adjusted Mutual Fund Performance (1996) — link
- Leakage and the Reproducibility Crisis in ML -based Science (2022) — link
- The secret sharer: Evaluating and testing unintended memorization in neural networks (2019)
- Quantifying memorization across neural language models (2022) — link
- Documenting Large Webtext Corpora : A Case Study on the Colossal Clean Crawled Corpus (2021) — link
- Time Travel in LLMs : Tracing Data Contamination in Large Language Models (2024) — link
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023) — link
- Training Data Leakage Analysis in Language Models (2021) — link
- Proving Test Set Contamination in Black Box Language Models (2023) — link
- Surveying Generative AI ’s Economic Expectations (2023) — link
- AI recognition of patient race in medical imaging: a modelling study (2022) — link
- Does Finance Benefit Society ? A Language Embedding Approach (2022) — link
- BloombergGPT : A Large Language Model for Finance (2023) — link
- In- Context Unlearning : Language Models as Few Shot Unlearners (2024) — link
- Time is Encoded in the Weights of Finetuned Language Models (2023) — link
- The promise and peril of generative AI : Evidence from ChatGPT as sell-side analysts (2023)
- Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
- Approaching Human - Level Forecasting with Language Models (2024) — link
- hmBERT : Historical Multilingual Language Models for Named Entity Recognition (2022) — link
- MacBERTh : Development and Evaluation of a Historically Pre -trained Language Model for English (1450-1950) (2021) — link
- Synthetic dataset generation techniques: generating custom sentence similarity data (2024) — link
- Yahoo! for Amazon : Sentiment Extraction from Small Talk on the Web (2007) — link
- Giving Content to Investor Sentiment: The Role of Media in the Stock Market (2007) — link
- More Than Words: Quantifying Language to Measure Firms’ Fundamentals (2008) — link
- How news and its context drive risk and returns around the world (2019) — link
- Predicting Returns With Text Data (2019)
- The colour of finance words (2023) — link
- Time Variation in the News – Returns Relationship (2023) — link
- Can chatgpt decipher fedspeak (2023) — link
- Memorization Without Overfitting : Analyzing the Training Dynamics of Large Language Models (2022) — link
- Extracting training data from large language models (2021)
- Upstream Mitigation Is \textit N ot All You Need : Testing the Bias Transfer Hypothesis in Pre - Trained Language Models (2022) — link
- Expected Returns and Large Language Models (2023) — link
- Distributed representations of words and phrases and their compositionality (2013)
- Glove: Global vectors for word representation (2014)
- Attention is all you need (2017)
- Language models are unsupervised multitask learners (2019)
- How to Learn and Teach Economics with Large Language Models , Including GPT (2023) — link
- Experimental evidence on the productivity effects of generative artificial intelligence (2023) — link
- Hellaswag: Can a machine really finish your sentence? (2019)
- ChatGPT, Stock Market Predictability and Links to the Macroeconomy (2023)
- Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference (2024) — link
- MosaicBERT : A bidirectional encoder optimized for fast pretraining (2023)
- BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- Textbooks are all you need (2023) — link
- The fineweb datasets: Decanting the web for the finest text data at scale (2024) — link
- Scaling data-constrained language models (2023)
- GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2019) — link
- FinBERT : A large language model for extracting information from financial text (2023)
- Robust performance hypothesis testing with the Sharpe ratio (2008)
- The Natural Language of Finance (2025)
- modded-nanogpt: Speedrunning the NanoGPT baseline (2024) — link
- Build NanoGPT (2024) — link
- Training compute-optimal large language models (2022) — link
- Pervasive underreaction: Evidence from high-frequency data (2021)
- Empirical Asset Pricing with Probability Forecasts (2024)
- Large language models: An applied econometric framework (2025)
- The Value of Information from Sell-side Analysts — → 2411.13813
- News implied volatility and disaster concerns (2017)
- Text selection (2021)
- Does finance benefit society? A language embedding approach (2025)
- Business news and business cycles (2024)
- When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks (2011)
- The llama 3 herd of models (2024) — link
- Efficient capital markets (1970)
- Why is it okay to average embeddings? (2020)
- HellaSwag: Can a Machine Really Finish Your Sentence? (2019) — link
- Re(Visiting) Large Language Models in Finance (2024)
- Caution Ahead : Numerical Reasoning and Look-ahead Bias in AI Models (2024) — link
- Crash narratives (2022)
- Entity Neutering (2025)
- The Memorization Problem : Can We Trust LLMs ’ Economic Forecasts ? — → 2504.14765
- AI ’s Predictable Memory and Its Implications for Finance (2025) — link
4 of 85 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.