Two findings: Strategy A earns 0.4 bps per trade with t = 4.2 on two million observations. Strategy B earns 40 bps per trade with t = 1.8 on three hundred. Which is real? Both. Which matters? Neither question is answered by the t-stat — A’s edge disappears into half a spread, and B needs more data. Statistical and economic significance are different questions, and the field’s chronic error is answering the first while claiming the second.
Why the usual significance bar is broken in finance
The textbook t > 2 threshold assumes you ran one test. Nobody runs one test. Academia has published hundreds of return “factors”; each represents many unreported variations tried along the way. Test enough random signals and 5% clear t > 2 by construction — the factor zoo problem. The influential response (Harvey, Liu & Zhu’s cross-section work) concluded that, accounting for the field’s collective data mining, a new factor should clear t ≈ 3.0 or higher — and large-scale replication studies applying consistent methodology found that a substantial fraction of published anomalies fail even at conventional thresholds.
The individual-researcher version is worse, because your trial count is invisible: fifty parameter combinations tried is fifty tests, and your best backtest is the maximum of fifty draws. What the maximum of N draws looks like by luck alone has a formula — that’s the deflated Sharpe ratio, this guide’s quantitative sequel. The honest practice it implies: count and report your trials. A t of 2.5 from three pre-registered variants outranks a t of 3.5 from a 2,000-run grid search.
Compounding it: financial data violates the assumptions under the t-stat — overlapping observations, correlated bets counted as independent, fat tails, and regime dependence all inflate naive significance. Clustered errors and block bootstraps help; they rarely rescue a marginal result.
Economic significance: the four questions t-stats can’t answer
- Magnitude after costs. Take the effect size, subtract realistic costs for the asset class, multiply by achievable exposure. Many famous anomalies are statistically robust and economically extinct — real patterns, too small to trade.
- Capacity. An edge confined to micro-caps or thin books can be simultaneously true and irrelevant beyond $2M — the square-root law guarantees it.
- Accessibility. Does capturing it require shorting hard-to-borrow names, trading at the close auction at scale, or data nobody can buy? A true effect you cannot access prices at zero.
- Decay. Published effects shrink — the well-documented post-publication decay in factor research — so the relevant magnitude is the post-publication, post-cost one, which is also why reproducing older papers on fresh data is the cleanest out-of-sample test available.
The two-by-two that organizes every empirical claim
| Economically large | Economically small | |
|---|---|---|
| Statistically strong | The real thing — rare; check costs, biases, capacity, then reproduce it | “True but tiny” — academically interesting, a trap for practitioners; most surviving anomalies live here |
| Statistically weak | “Promising but unproven” — the interesting quadrant; gather data, don’t dismiss (small samples doom real effects too) | Noise — walk away regardless of narrative quality |
The top-right trap deserves emphasis because it’s where sophisticated readers get caught: a mountain of observations makes tiny effects gleam with enormous t-stats. High-frequency datasets are factories for this quadrant — two million ticks will find you a 0.3-bp effect at t = 6 that no fee schedule on earth lets you keep.
How this maps to our scores
The empirical-rigor axis on every paper page is, in large part, this guide operationalized: papers rank higher for reporting effect sizes with costs, disclosing trial counts, testing subperiods, and distinguishing what’s significant from what’s tradeable. Papers with spectacular t-stats and silent turnover columns rank lower — now you know exactly why.
The quantitative sequel: The deflated Sharpe ratio → · The sample-size discipline: walk-forward & OOS testing → · From the corpus: ARDL models: appropriate use vs p-hacking →