Six months from now, a result row says Sharpe 1.4 and you need to know: exactly what produced this number? If the answer requires memory, the result is folklore. Versioning is the discipline that makes every number re-derivable from code version × config version × data version — and in research (unlike app development) the data axis is the one that bites, because data changes under you: vendors restate, adjustments rewrite history, and your own ingestion pipeline evolves.
Versioning data: snapshots + content addresses
The workable pattern for file-tier data, without heavy tooling:
- Immutability by convention: partitions, once written, are never edited — corrections write new partitions (the raw zone’s append-only rule extended downstream). Mutation is the enemy of versioning; remove it and versioning reduces to naming.
- Content-address the snapshots: a manifest per dataset version — file list + sizes + checksums (a few lines of hashing) — stored in git next to the code.
data_version = manifest hash. Now “the data changed” is a diff, not a suspicion. - Derived data versions by recipe: features and bars are functions of (upstream version, transform code version); store that pair as their version and treat the outputs as regenerable cache. If regeneration doesn’t reproduce byte-identical outputs, you’ve found nondeterminism worth killing.
- The database tier versions differently: bitemporal tables version themselves (knowledge time is a version axis); for the rest, timestamped
pg_dumps you’ve actually restored.
Purpose-built tools (DVC, lakeFS, Delta) formalize all this; adopt one when manifests-in-git strains, not before — the concept, not the tool, is the requirement.
Versioning backtests: the run record
Every backtest run writes one row, at run time, automatically (the harness’s job, never a human’s):
run_id | git_commit (code) | config_hash (full resolved params)
| data_versions (manifest hashes consumed) | seed
| environment (lockfile hash) | started/finished | metrics | artifacts_path
Three details carry the weight: hash the resolved config (defaults filled in — “the default changed” is the classic silent result-drift); record data versions consumed, not “latest” (the run must name its snapshots, or reruns quietly use different data); and store the row even for failed and abandoned runs — the run table is also your trial counter, and dead runs count.
The lineage chain, end to end
The standard a reproduction of your own work should meet: any reported number traces to a run row → which names code commit, config hash, data manifests, seed, environment → each of which is fetchable → and rerunning reproduces the number to stated tolerance. That chain is what “reproducible” means operationally — and it’s perhaps two days of harness work, mostly bookkeeping. The payoff compounds: live-vs-backtest divergence debugging starts from “which exact inputs produced the research claim?”, answered in seconds.
Common versioning failures
- “Latest” as a data reference: the run that can never be rerun; every downstream comparison contaminated by data drift.
- Notebook results with no run row: the exploratory finding that can’t be re-derived — the moment a number matters, it goes through the harness.
- Config in code: parameters edited in place between runs; the git history is the config history only if configs live in files.
- Un-versioned environment: dependency drift changes results months later; the lockfile hash belongs in the run row.
- Adjustment-vintage amnesia: results computed on last year’s adjusted prices compared against this year’s — the data-version axis’s sneakiest instance.
Questions to ask (of your own shop, quarterly)
- Pick a number from three months ago: can you re-derive it, today, from its row?
- Do any pipelines read “latest”?
- Does the trial count in DSR calculations come from the run table or from memory?
- Would a new teammate’s rerun match yours byte-for-byte?
What our scoring captures
Papers can’t show their run tables, but the textual shadows of this discipline — stated seeds, pinned versions, config listings, code that reproduces specific tables — are strong rigor signals, and their absence in ML-heavy work is the genre’s reproducibility gap made visible. Internally, this guide is the difference between research and anecdote generation.
The process around it: research pipeline → · The environment axis: Docker reproducibility → · The experiment layer: ML tracking stack →