Paper: arXiv 2611.03985

Authors: Amir Dehkordi, Alireza Kargarzadeh, Nariman Khaledian, Navid Parvini, Sid Ghatak, Arman Khaledian

Abstract

Large language models (LLMs) read company news at scale, and backtests suggest that their readings predict stock returns. We ask how much of that predictability survives a test that rules out look-ahead, memorization and selection. We propose OLIVIA, a protocol for such tests: point-in-time inputs with company identities masked, selection on validation data only, an analysis plan frozen by cryptographic hash, and a test period unblinded once through a logged lock. We apply it to AI decision models, non-generative language models that return probabilities for typed questions, reading news about S&P 500 stocks: an open-weights model we fine-tuned on 2024-2025 data, three commercial models and standard baselines. The fine-tuned model’s signal replicated on two validation quarters (daily information coefficients 0.055 and 0.065) but fell to 0.020 (t = 1.7) on the blinded January-July 2026 test. The commercial models’ readings line up with the overnight move to the open (information coefficient 0.074 to 0.077) but not with the move after it, so the news is priced by the time a trader can act on it at the open. No model can tell over- from under-reaction, and no pre-registered strategy survives costs and a deflated Sharpe ratio. Commercial models identify masked companies in 62% to 74% of cases against a 28% baseline: masking hides names, not companies. The market reads the news first.

Complexity vs Empirical Score

  • Math Complexity: 6.0/10
  • Empirical Rigor: 9.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper demonstrates high empirical rigor through its robust OLIVIA protocol for testing LLM signals, including pre-registration and blinded tests. While the mathematical complexity is moderate, the rigorous statistical methodology and novel testing framework elevate its overall quality significantly.

Research Flowchart

  flowchart TD
    A[Research Goal: Do LLM News Readings Predict Stock Returns After Robust Testing?] --> B{Key Methodology: OLIVIA Protocol};
    B --> C[Data/Inputs: S&P 500 News, Point-in-time, Masked Identities];
    C --> D[Computational Processes: Fine-tuned LLM, Commercial LLMs, Baselines, IC Calculation, Strategy Backtesting];
    D --> E{Key Finding 1: Fine-tuned LLM signal replicated on validation, but weakened on blinded test (IC=0.020)};
    E --> F{Key Finding 2: Commercial LLMs predict overnight open move (IC=0.074-0.077), but not post-open move};
    F --> G{Key Finding 3: No model identifies over/under-reaction; no pre-registered strategy survives costs/deflated Sharpe. Masking hides names, not companies. Market reads news first.};