Most impressive backtests are wrong, and most are wrong unintentionally — which is worse, because the author’s confidence is genuine. After scoring thousands of empirical papers for rigor, the failures cluster into eight questions. Ask them in order; each has a verdict rule. A backtest that survives all eight isn’t proven — but one that fails two or more is not evidence of anything.
1. Could the strategy have known its own universe?
Survivorship is the first check because it silently manufactures alpha before any signal is computed. The universe at each historical date must be “securities tradeable on that date,” including everything that later delisted, defaulted, or was absorbed. Verdict rule: if the paper doesn’t name a survivorship-clean source or describe delisting handling, assume 1–4%/yr of phantom performance on long equity strategies — enough to erase most published edges. Full treatment: Survivorship bias →
2. Could the signal have known the future?
Look-ahead leaks arrive through restated fundamentals, final index membership, same-bar execution, and — increasingly — ML models trained on text corpora that include the test period. Verdict rule: delay every signal by one bar and rerun. Alpha that halves under a one-bar lag was living on timing leakage; alpha that vanishes was never there. Full taxonomy: Look-ahead bias & point-in-time data →
3. What happens at 2× the assumed costs?
Every backtest embeds a cost assumption, often zero. The question isn’t whether the assumption is right — it’s how fragile the result is to being wrong. Verdict rule: demand (or rebuild) the cost-sensitivity curve at 1×, 2×, 3× assumed costs, as required by Gate 1 of our production checklist. High-turnover strategies that die at 2× costs are cost models wearing a strategy costume. The cost arithmetic itself: Transaction costs & market impact →
4. How many independent bets is this, really?
Four hundred trades across twelve correlated event clusters is twelve observations. Fifty markets on one election is one bet (the prediction-market version). Overlapping multi-day holding periods shrink effective sample size the same way. Verdict rule: cluster trades by underlying event/period before computing any significance; if the paper reports per-trade statistics on overlapping positions without adjustment, discount its t-stats heavily. Then apply the multiple-testing lens — how many variants were tried to find this one?
5. Does it survive the neighborhood of its own parameters?
A real edge is a plateau; an artifact is a spike. Lookback 20 works, lookbacks 15 and 25 collapse → you’re looking at noise that happened to fit. Verdict rule: ask for (or grid) the parameter neighborhood. Papers that report a single parameterization with no sensitivity analysis earn a “Lab Rats”-adjacent discount on our rigor axis regardless of how good the single number looks.
6. Was out-of-sample actually out of sample?
The phrase “out-of-sample” does heavy, often dishonest work. A holdout that was evaluated repeatedly during development is training data with extra steps. Verdict rule: look for the protocol, not the split — walk-forward structure, embargo periods, and evidence the holdout was touched once. The strongest tell of honesty is a paper reporting worse OOS than in-sample numbers and discussing why. Full protocol: Walk-forward & out-of-sample testing →
7. Would the market let you do this at size?
A 3% edge on a universe trading $200k/day is a hobby, not a strategy. Verdict rule: multiply implied position sizes against real ADV and book depth; anything above ~1–5% of daily volume needs an impact model, not a fee assumption. Papers on small caps, crypto long-tails, and prediction markets fail here constantly — the phantom-liquidity problem.
8. Is the reporting complete enough to be falsifiable?
The minimum honest panel: returns and turnover and max drawdown and exposure over time and subperiod breakdown and the number of strategies tried. Each omission hides a specific failure mode (no turnover → cost fragility; no subperiods → one-regime wonder; no trial count → p-hacking). Verdict rule: treat missing panels as adverse inference — authors publish what flatters.
The meta-test: luck has a distribution
Any single equity curve — however smooth — is one draw from a fan of possible paths. Before believing a backtest, ask what the fan looks like: our equity-curve simulator shows how wide the outcome distribution is for a strategy with a genuinely fixed edge, and the deflated Sharpe ratio quantifies how impressive the best of N attempts should look by luck alone. A backtest is real when it clears that bar — not when it merely looks good.
The 60-second version
| Question | Kill criterion |
|---|---|
| Universe | No survivorship handling stated |
| Timing | Alpha dies at one-bar lag |
| Costs | Dead at 2× assumed costs |
| Sample | Correlated bets counted as independent |
| Parameters | Spike, not plateau |
| OOS | No protocol, or reused holdout |
| Capacity | Size > ~5% ADV with no impact model |
| Reporting | Missing turnover/drawdown/trials |
Apply it: highest-rigor papers pass most of this by construction · reproduce one yourself → · From the corpus: trading simulation: replay vs agent-based vs generative →