Paper: arXiv 2602.00133

Authors: Avi Arora, Ritesh Malpani

Abstract

Prediction markets offer a natural testbed for trading agents: contracts have binary payoffs, prices can be interpreted as probabilities, and realized performance depends critically on market microstructure, fees, and settlement risk. We introduce PredictionMarketBench, a SWE-bench-style benchmark for evaluating algorithmic and LLM-based trading agents on prediction markets via deterministic, event-driven replay of historical limit-order-book and trade data. PredictionMarketBench standardizes (i) episode construction from raw exchange streams (orderbooks, trades, lifecycle, settlement), (ii) an execution-realistic simulator with maker/taker semantics and fee modeling, and (iii) a tool-based agent interface that supports both classical strategies and tool-calling LLM agents with reproducible trajectories. We release four Kalshi-based episodes spanning cryptocurrency, weather, and sports. Baseline results show that naive trading agents can underperform due to transaction costs and settlement losses, while fee-aware algorithmic strategies remain competitive in volatile episodes.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.5: no out-of-sample test; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper introduces a novel, highly practical event-driven benchmark framework for backtesting AI and algorithmic trading agents on prediction markets. While mathematical theoretical development is minimal, the empirical setup is rigorous, execution-realistic, and highly reproducible with provided open-source code and detailed historical datasets.

Research Flowchart

  flowchart TD
    A["Research Goal: Benchmark LLM & Algorithmic Trading Agents on Prediction Markets"] --> B["Data Inputs: Raw Kalshi Streams<br/>(Orderbooks, Trades, Lifecycle, Settlement)"]
    B --> C["PredictionMarketBench Processing<br/>(Episode Construction & Tool-Based Interface)"]
    C --> D["Execution-Realistic Simulation<br/>(Event-Driven Replay, Fees, Maker/Taker Semantics)"]
    D --> E["Strategy Evaluation<br/>(Classical vs. Tool-Calling LLM Agents)"]
    E --> F["Key Findings: Naive agents lose to fees/settlement;<br/>Fee-aware strategies stay competitive"]

Cited References (18)

  1. Prediction Markets (2004)
  2. SWE-bench : Can Language Models Resolve Real-World GitHub Issues? (2024)
  3. The Probability of Backtest Overfitting (2017)
  4. How is Kalshi regulated?
  5. Maker-taker
  6. The Promise of Prediction Markets (2008)
  7. Combinatorial Information Market Design (2003)
  8. Market Microstructure Theory (1995)
and 10 more
  1. CFTC Issues Order Granting Kalshi EX Designation as a Contract Market (2020)
  2. Bollinger on Bollinger Bands (2001)
  3. Prediction market accuracy in the long run (2008)
  4. Market Design, Manipulation, and Accuracy in Political Prediction Markets: Lessons from the Iowa Electronic Markets (2014)
  5. Agent-Based Modeling of the Prediction Markets (2011)
  6. Semantic Trading: Agentic AI for Clustering and Relationship Discovery in Prediction Markets (2025)
  7. Reinforcement Learning in Limit Order Markets (2019)
  8. FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021)
  9. ABIDES: Towards High-Fidelity Market Simulation for AI Research (2019)
  10. Multi-Agent Reinforcement Learning in a Realistic Limit Order Book Market Simulation (2020)

0 of 18 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.