Prediction markets look like the easiest possible backtesting target: prices are probabilities, outcomes are binary, resolution is objective. That surface simplicity is exactly why most published prediction-market “edges” evaporate on contact with a live order book. The failure modes here are different from equities, and equity-quant reflexes miss them. This guide is the methodology we’d hold any strategy to before real capital touches a CLOB — written from the perspective of someone who has watched paper edges die in production.

Why prediction markets break equity intuitions

Four structural differences drive everything below:

  1. Terminal, binary payoffs. Every position resolves to 0 or 1 by a known-ish date. There’s no “hold and be right eventually” — timing and capital lockup are part of the bet.
  2. Prices are probabilities, so calibration analysis — not just P&L — is available and mandatory.
  3. Books are thin. Outside headline politics/sports markets, visible depth is often a few hundred to a few thousand dollars within a cent of mid. Any backtest that fills meaningful size at mid is fiction.
  4. Markets are born and die constantly, and the historical record you can easily download is dominated by markets that resolved cleanly — a survivorship filter nobody advertises.

The five biases, in the order they’ll get you

1. Resolution leakage (look-ahead’s nastier cousin)

The classic error: your feature set knows things that were only knowable near resolution. It sneaks in through innocuous routes — using a market’s final metadata (titles and descriptions get edited), computing features over a window that overlaps the resolution announcement, or training an NLP model on news text whose corpus extends past the event. The discipline: every feature must be computable from a snapshot taken strictly before the trade timestamp, and market metadata must be the creation-time version. If you can’t reconstruct point-in-time metadata, exclude the feature.

2. Phantom liquidity

Prediction-market backtests are usually built on trade prints or last-price series, then evaluated as if you could trade at those prices in your size. In thin books this overstates capacity by an order of magnitude. Minimum honest fill model:

  • Cross the spread: buys lift the ask, sells hit the bid — never fill at mid or last.
  • Cap fill size at a fraction (say a third) of visible depth at that level, or walk the book if you have depth data.
  • Apply the platform’s actual fee schedule (taker fees, settlement fees, and on-chain venues, gas) per fill, not per backtest.

Record your own books. Most venues do not provide historical order-book depth; trade prints are not depth. A cron job snapshotting the CLOB for your candidate markets every few minutes costs almost nothing and is the single highest-value data asset a prediction-market trader can own. Start recording before you need the history — nobody sells it to you later.

3. Survivorship and the void-market problem

Resolved-market datasets silently exclude or mishandle markets that were voided, resolved “N/A,” got disputed, or had their terms clarified mid-life. Strategies harvesting “cheap longshots” or “expensive favorites” are especially distorted: disputed and voided markets cluster exactly in the weird tails where those strategies live. Your universe at each point in time must be “markets tradeable at time t,” including the ones that later resolved ugly — and dispute/void outcomes must hit the P&L (usually as a refund minus fees and a lot of locked capital time).

4. Correlated event clusters

Fifty markets on one election are one bet wearing fifty costumes. Naive per-market analysis reports n=50 independent wins when the true sample size is 1. Cluster by underlying event before computing any statistics; a strategy with 400 trades across 12 event clusters has 12 data points for significance purposes. This is the prediction-market version of the multiple-testing problem that wrecks factor research.

5. Capital lockup and the funding illusion

A 4% edge that locks capital for nine months is a worse trade than a 1% edge that resolves weekly. Backtests that report return-per-trade without annualizing per unit of locked capital systematically favor long-dated markets. Track capital-days per position and report return on committed capital per unit time — it reorders most strategy leaderboards.

Calibration before P&L

Before any profit claim, run the sanity layer that binary markets uniquely allow:

  • Reliability curve: bucket your model’s predicted probabilities, plot realized frequencies. A model that says 70% should be right ~70% of the time.
  • Brier score / log loss against two baselines: the market price itself, and a naive “always the market price minus fees” model. If your model doesn’t beat the market’s own calibration, your edge is fee-negative noise.
  • Edge decomposition: is the P&L from probability estimation, from liquidity provision (earning spread), or from timing flows? These have completely different capacity and decay profiles, and mixing them in one backtest number hides which one you actually have.

The walk-forward protocol

  1. Universe construction in event time: at each historical date, list markets that existed, were tradeable, and met your liquidity floor using only that date’s information.
  2. Strict temporal splits for any learned model, with embargo periods around cluster boundaries (train on 2023–24 politics, test on 2025+ — never interleave markets from one event across splits).
  3. Fee-and-spread-first reporting: publish the gross-to-net waterfall. In our experience the spread+fee haircut on thin markets routinely consumes 60–100% of apparent gross edge.
  4. Paper trading against live books for 4–8 weeks minimum, with your real execution code, comparing realized fills to backtest-assumed fills. The fill-quality gap is your ongoing reality-check metric, not a one-time gate.
  5. Tiny-size live period before scale, per the research-to-production checklist.

Tooling that makes this tractable

  • Venue CLOB/data APIs for live books and trade prints; your own snapshot archive for depth history (Parquet + DuckDB handles years of it — see the stack guide).
  • Resolution-source archives (the oracle/UMA record, official statistics pages) with timestamps, so resolution timing itself is a modeled variable.
  • A position/risk monitor with per-event exposure caps — the correlated-cluster bias exists in live trading too, not just backtests.

We’re packaging this methodology as a runnable backtest template (notebook + fill model + calibration suite). Subscribers hear first when it ships — signup below.

Related: Crypto & DeFi research hub → · Production checklist →