Experiment tracking in mainstream ML answers “which config was best?” In quant research it must answer a second, harder question: “how many things did we try?” — because the trial count is a statistical input, and an untracked experiment is both irreproducible and an uncounted draw against your luck budget. That dual purpose shapes everything below.
What a run record must contain
The versioning guide’s run row is the base: code commit, resolved-config hash, data-manifest versions, seed, environment digest, metrics, artifacts. Quant ML adds fields generic tools don’t prompt for:
- Split protocol, recorded as data: the exact walk-forward/purge/embargo scheme — fold boundaries and embargo widths as stored values, so “was this leaked?” is a query, not archaeology.
- Holdout-touch ledger: which runs read the final test set. The evaluate-once rule is only enforceable if touches are logged.
- Experiment lineage / family: which idea-log entry a run serves, and its parent run — the structure that makes N countable per claim, which is what DSR and CSCV actually consume.
- Economic metrics beside ML metrics: accuracy and costed P&L translation, turnover, per-fold Sharpe — recorded per run so the accuracy-vs-P&L divergence is visible in the tracker, not discovered later.
- Seed batch, not seed: multi-seed dispersion as a first-class result; a run family with one seed is flagged incomplete.
The minimal stack
Tier 0 — one table, one decorator (where most solo shops should stay): a runs table in your existing Postgres, a @tracked decorator that snapshots config/commit/data-versions before and metrics after, artifacts to dated directories. ~150 lines, zero services, queryable with SQL — and because it’s your schema, the finance fields above are just columns. This is the experiment log and run registry unified.
Tier 1 — MLflow (self-hosted): adds a UI, artifact management, and model registry for the cost of one more container. Adopt when run volume makes SQL browsing tedious or a second person arrives. Keep the finance fields as tags/params — the schema discipline stays yours.
Tier 2 — W&B/hosted: collaboration, sweeps, dashboards — real value at team scale; at solo scale mostly a subscription and a data-residency question (strategy configs are IP; check terms before they live on someone’s cloud).
The anti-pattern at every tier: tracking as UI, not as constraint. The tracker earns its keep when the backtest harness refuses untracked runs — the enforcement, not the dashboard, is the product.
Sweeps: where tracking meets statistics
Hyperparameter sweeps are industrialized trial generation, so the sweep config itself is a research artifact: record the grid searched (not just winners), sweep-level N, and selection rule — then report the winner deflated by that N, with the parameter-plateau view saved as a sweep artifact. A sweep whose losers were deleted is a p-hacking machine with good UX.
Common tracking failures
- Notebook runs off the books: the exploratory phase generates most trials and zero rows — the moment a number matters, it goes through the harness.
- Deleted failures: cleaning “clutter” destroys the denominator; failed runs are load-bearing data.
- Metrics without splits: a tracked Sharpe with an untracked split protocol can’t be audited for leakage.
- Tracker-as-optional: two paths through the codebase (tracked and quick) — the quick one wins, and the log lies.
Questions to ask (of your setup, or a paper’s)
- Can you compute an honest N per headline claim, from records?
- Are holdout touches logged and countable?
- Do failed and abandoned runs persist?
- Could a colleague rerun the best experiment from its row alone?
What our scoring captures
Papers rarely show trackers, but their shadows — disclosed search spaces, per-seed dispersion, split protocols stated as data — are among the rigor axis’s strongest ML signals, and their absence is why the ML hub pairs high volume with high skepticism. Internally: this stack is what makes your own claims scoreable by your own standards.
The statistics it feeds: deflated Sharpe → · CSCV → · The pipeline home: research pipeline → · The compute: GPU decisions →