A paper you’ve read is a story; a paper you’ve reproduced is a result. The gap between the two is where most published alpha lives — and dies. This is the checklist we use when a paper survives triage and the backtest looks real: six stages, each with its characteristic trap. A condensed version is downloadable at the end.

Stage 1 — Acquire: establish what exists

  • ☐ Paper version: get the latest revision (arXiv papers mutate; results sometimes soften between v1 and v4 — diff the tables).
  • ☐ Code: linked repo? Note the commit hash and whether it actually produces the paper’s tables or is “illustrative.” Most repos are the latter.
  • ☐ Data: can you buy or download the same data — same vendor, frequency, adjustments? “CRSP 1963–2018” is reproducible; “proprietary tick data” ends the exercise honestly (mark it unreproducible, not wrong).
  • ☐ Compute budget: estimate before starting. A 5-minute daily-bars study and a GPU-month of deep-model training are different commitments — price it first.

The trap: starting the reimplementation before confirming the data is obtainable. Data access decides reproducibility more often than method complexity does — which is why vendor choice is a research-methodology issue, not procurement.

Stage 2 — Align: match their world before testing it

  • ☐ Universe construction rules (exchanges, price floors, liquidity filters, share classes) reproduced as of each date — survivorship-clean.
  • ☐ Sample dates exactly; note anything the window conveniently excludes (2008? 2020? 2022?).
  • ☐ Definitions pinned: “monthly return” (close-to-close? which day?), “volatility” (realized? which estimator?), “momentum” (12-1? 6-1?). Every undefined term is a fork in the road.
  • ☐ Adjustment policy: splits, dividends, restatements — matched to what the paper’s vendor would have shown at the time.

The trap: silently using your own conventions. When your numbers diverge later, you won’t know if the paper is wrong or your alignment is.

Stage 3 — Reimplement: independently, on purpose

  • ☐ Write the pipeline from the paper’s description before reading their code. Their code is the answer key, and answer keys teach nothing when copied — worse, porting their code ports their bugs. (Our T-KAN replication review found the headline comparison rested on a baseline whose LSTM was never trained — invisible unless you reimplement or audit, not copy.)
  • ☐ One reproducible environment: pinned dependencies, fixed seeds, logged config — the same discipline as taking a strategy live.
  • ☐ Where the paper is ambiguous, log the decision and both interpretations. The ambiguity list is a research artifact.

Stage 4 — Reconcile: match statistics before results

The order matters enormously and everyone gets it backwards:

  • ☐ First match descriptive statistics: universe size per year, mean/vol of returns, coverage. If your universe has 2,900 stocks and theirs 3,400, stop — nothing downstream is interpretable.
  • ☐ Then match intermediate artifacts: signal distributions, factor loadings, portfolio turnover.
  • ☐ Only then compare headline results — with tolerance bands agreed in advance (sign and rough magnitude are success; the third decimal is not).
  • ☐ Feed the pipeline a case with a known answer and demand machine precision — the strongest validation pattern we know, per the Monte Carlo audit.

The trap: jumping straight to the Sharpe ratio. When it doesn’t match, you now have one uninterpretable number and no idea which of forty upstream choices caused it.

Stage 5 — Stress: the part the paper didn’t do

Reproduction confirms the claim as stated; stress testing asks whether it matters:

Stage 6 — Document: make your reproduction reproducible

  • ☐ A short report: what matched, what diverged, the ambiguity list, the stress results.
  • ☐ Publish it if you can — replication notes are among the most valuable, least supplied artifacts in the field (and exactly what the discussion sections on our paper pages exist for).
  • ☐ Archive data snapshots and environment — per the raw-zone discipline.

The honest outcome distribution

Expect roughly: a third of attempts reproduce cleanly, a third reproduce weaker (the median published effect shrinks out of the author’s hands), and a third fail on data access, ambiguity, or genuine error. All three outcomes are wins — the failed reproduction that costs you two weeks is the strategy that would have cost you a drawdown.

📥 Take it with you: download the condensed checklist (.md) — drop it into the repo of your next reproduction attempt.

Companion frameworks: Is the backtest real? → · Auditing a Monte Carlo →