Definition. An ML trading paper claims a learned model — trees, LSTMs, transformers, anything with a loss function — predicts returns, prices, or market states well enough to matter. It is the largest genre in our archive and the one where the gap between reported and real performance is widest, because ML adds new failure channels on top of the standard backtest sins.
The worked example: the accuracy illusion
A paper reports 57% directional accuracy on daily S&P moves. Sounds tradeable. Now the arithmetic: with daily vol ~1%, a 57/43 edge on direction — if real — yields a gross Sharpe near 2, but three deductions apply before belief. (1) The base rate: the index rises ~53% of days, so a constant “up” prediction already scores 53% — the paper’s edge over naive is 4 points, not 7. (2) Accuracy weights all days equally; P&L doesn’t — models routinely get the small moves right and the crashes wrong, so accuracy can rise while returns fall. (3) The number is one seed, one split. Rerun with five seeds and the spread often covers the entire claimed edge. Any paper reporting accuracy without a naive-baseline comparison, a P&L translation, and seed variance has reported approximately nothing.
The ML-paper checklist
- ☐ Baselines that can win: logistic regression, gradient-boosted trees on the same features, and buy-and-hold. A deep model beating only other deep models is an ablation. Trees-on-tabular remains the bar to clear in this field — a result that recurs across our hub.
- ☐ Pipeline-level leakage audit: scalers fit on the full sample, features selected by full-sample correlation, labels overlapping the train/test boundary, pretrained models with post-test-period knowledge. ML leakage lives in the pipeline, not the dataset.
- ☐ Temporal splits with purging/embargo — random k-fold on market data is disqualifying, full stop.
- ☐ Seeds and stability: multiple seeds with dispersion reported; single-run deep-learning results in finance are anecdotes.
- ☐ Hyperparameter accounting: the search grid is a trial count; tuned-on-test is the genre’s quiet epidemic.
- ☐ Economic translation: predictions → portfolio → costs → net Sharpe. Papers stopping at RMSE or accuracy have tested a model, not a strategy.
- ☐ Was the baseline trained correctly? Verify, don’t assume — our T-KAN replication review found a headline comparison whose baseline LSTM was never trained at all.
Questions to ask while reading
- What does the naive baseline score, and is the gap economically meaningful after costs?
- Could I reproduce the data pipeline from the paper alone — or only the architecture?
- How many configurations were evaluated, and where did the test set enter that loop?
- Does performance concentrate in a subperiod, a volatility regime, or a handful of names?
- If the model is interpretable-by-claim, do the shown “learned” artifacts actually change with training? (We’ve caught fixed activation functions labeled as learned.)
What our scoring captures — and misses
ML papers cluster in our “Holy Grail” and “Lab Rats” quadrants: the rigor axis rewards real out-of-sample protocol and cost accounting, and it demotes the accuracy-only genre effectively. What the score can’t verify from a paper’s text is pipeline leakage — a paper can describe a clean protocol and implement a dirty one. That’s only catchable at the code level, which is why ML papers are the genre where reproduction pays the highest information dividend per hour spent.
Companions: Evaluating RL trading papers → · Walk-forward & OOS → · Backtest overfitting → · From the corpus: XGBoost in finance: what the papers show →