Forecasting Future Language: Context Design for Mention Markets

Paper: arXiv 2602.21229

Authors: Sumin Kim, Jihoon Kwon, Yoon Kim, Nicole Kagan, Raffi Khatchadourian, Wonbin Ahn, Alejandro Lopez-Lira, Jaewon Lee, Yoontae Hwang, Oscar Levy, Yongjae Lee, Chanyeol Choi

Abstract

Mention markets, a type of prediction market in which contracts resolve based on whether a specified keyword is mentioned during a future public event, require accurate probabilistic forecasts of keyword-mention outcomes. While recent work shows that large language models (LLMs) can generate forecasts competitive with human forecasters, it remains unclear how input context should be designed to support accurate prediction. In this paper, we study this question through experiments on earnings-call mention markets, which require forecasting whether a company will mention a specified keyword during its upcoming call. We run controlled comparisons varying (i) which contextual information is provided (news and/or prior earnings-call transcripts) and (ii) how \textit{market probability}, (i.e., prediction market contract price) is used. We introduce Market-Conditioned Prompting (MCP), which explicitly treats the market-implied probability as a prior and instructs the LLM to update this prior using textual evidence, rather than re-predicting the base rate from scratch. In our experiments, we find three insights: (1) richer context consistently improves forecasting performance; (2) market-conditioned prompting (MCP), which treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and (3) a mixture of the market probability and MCP (MixMCP) outperforms the market baseline. By dampening the LLM’s posterior update with the market prior, MixMCP yields more robust predictions than either the market or the LLM alone.

Complexity vs Empirical Score

• Math Complexity: 2.5/10
• Empirical Rigor: 7.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper presents an empirically strong and novel framework for forecasting keyword mentions in prediction markets (Kalshi) by combining LLM context processing with market-implied probability priors (MCP and MixMCP). Math is minimal and relies primarily on basic probability mixture rules and standard scoring rules (Brier, ECE), but the empirical evaluation on real prediction market data is rigorous and well-designed.

Research Flowchart

  flowchart TD
    A[Research Goal: Optimize LLM Context & Prompting for Mention Market Forecasting] --> B[Data Inputs: News Articles, Prior Earnings Call Transcripts, Market Probabilities]
    B --> C[Methodology: Controlled Experiments on Earnings Call Mention Markets]
    C --> D[Prompting Approaches: Market-Conditioned Prompting MCP vs MixMCP Hybrid]
    D --> E[Key Outcomes: Richer Context Improves Performance, MCP Calibrates Forecasts, MixMCP Outperforms Baseline Market]

Cited References (14)

1. Prediction Markets (2004) — link
2. The Promise of Prediction Markets (2008) — link
3. Approaching human-level forecasting with language models (2024) — link
4. AIA Forecaster: Technical Report (2025) — link
5. Earnings conference calls and stock returns: The incremental informativeness of textual tone (2012)
6. Say it again Sam: the information content of corporate conference calls (2017)
7. Verification of forecasts expressed in terms of probability (1950)
8. Strictly Proper Scoring Rules, Prediction, and Estimation (2007) — link

0 of 14 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.