PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

Paper: arXiv 2604.14199

Authors: Pu Cheng, Juncheng Liu, Yunshen Long

Abstract

Predicting real-world events from live market signals demands systems that fuse qualitative news with quantitative order-book dynamics under strict temporal discipline – a challenge existing benchmarks fail to capture. We present \textbf{PolyBench}, a multimodal benchmark derived from Polymarket that records point-in-time cross-sections of 38,666 binary prediction markets spanning 4,997 events, synchronously coupling each snapshot with a Central Limit Order Book (CLOB) state and a real-time news stream. Using PolyBench, we evaluate seven state-of-the-art Large Language Models – spanning open- and closed-source families – generating 36,165 predictions under identical, timestamp-locked market states collected between February 6 and 12, 2026. Our multidimensional framework assesses directional accuracy, our proposed Confidence-Weighted Return (CWR), Annualized Percentage Yield (APY), and Sharpe ratio via realistic order-book execution simulation. The results reveal a pronounced performance divergence: only two of seven models achieve positive financial returns – MiMo-V2-Flash at \textbf{17.6%} CWR and Gemini-3-Flash at 6.2% CWR – while the remaining five incur losses despite uniformly high stated confidence. These findings highlight the gap between surface-level language fluency and genuine probabilistic reasoning under live market uncertainty, and establish PolyBench as a contamination-proof, financially-grounded evaluation standard for future LLM research. Our dataset and code available at \underline{\href{https://github.com/PolyBench/PolyBench}{https://github.com/PolyBench/PolyBench}}.

Complexity vs Empirical Score

• Math Complexity: 3.5/10
• Empirical Rigor: 8.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: PolyBench provides a highly rigorous, contamination-proof benchmark evaluating LLMs on live prediction market order books and news streams. While the underlying mathematical formulations are straightforward (financial performance metrics and execution simulations), the empirical methodology, dataset size, and open-source availability make it a strong, practical contribution.

Research Flowchart

  flowchart TD
    A["Research Goal: Evaluate LLM forecasting & trading on live prediction market data"] --> B["Data Collection: PolyBench Dataset<br/>(38,666 markets, 4,997 events, CLOB states & real-time news)"]
    B --> C["Experimental Setup: Test 7 SOTA LLMs<br/>(36,165 predictions under timestamp-locked states)"]
    C --> D["Simulation & Evaluation Framework<br/>(Simulated order-book execution & CWR, APY, Sharpe ratio metrics)"]
    D --> E["Key Outcomes & Findings"]
    
    E --> F["Performance Divergence:<br/>Only 2/7 models profitable<br/>(MiMo-V2-Flash: +17.6% CWR, Gemini-3-Flash: +6.2% CWR)"]
    E --> G["Core Insight:<br/>5 models suffered losses despite high confidence,<br/>revealing a gap in probabilistic reasoning"]

Cited References (88)

1. Time series analysis: Forecasting and control (1970)
2. Generalized autoregressive conditional heteroskedasticity (1986)
3. Long short-term memory (1997)
4. Attention is all you need (2017)
5. BloombergGPT: A large language model for finance (2023) — link
6. FinBERT: Financial sentiment analysis with pre-trained language models (2019) — link
7. Time-LLM: Time series forecasting by reprogramming large language models (2023) — link
8. LiveTradeBench: Seeking Real-World Alpha with Large Language Models (2025) — link

0 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.