Definition. P-hacking is the family of practices that manufacture statistical significance through flexibility: adjusting samples, specifications, and metrics after seeing results until p < 0.05 appears. In finance it is rarely fraud and usually drift — each choice locally defensible, the sequence jointly fatal. It’s the human-behavior layer beneath backtest overfitting (the selection mechanics) and the multiple-testing problem (the field-level arithmetic); this guide is about recognizing the practices — in your own work and in the papers you read.
The seven practices, in ascending order of self-deception
- Outcome switching — the study set out to test monthly returns; weekly “turned out to be more appropriate.”
- Sample sculpting — dropping “anomalous” periods (2008, 2020, that one bad year), filtering “illiquid” names after seeing they hurt, or the start date that happens to flatter.
- Specification shopping — trying controls, weightings, and breakpoints until significance; reporting the winner as the specification (the factor-paper version).
- Covert stopping rules — extending or truncating the sample until the result stabilizes where wanted.
- Metric shopping — Sharpe didn’t clear the bar, but Sortino did; accuracy failed, but F1 passed (the ML variant).
- Subgroup mining — the effect “concentrates” in small caps / January / high-vol regimes, discovered after the pooled test failed. Sometimes real; always suspect when unplanned.
- HARKing — Hypothesizing After Results are Known: the mechanism story written last, presented first. The tell is a theory section that predicts exactly the estimated coefficients and nothing else.
The numerical intuition
The flexibility arithmetic is unforgiving. Suppose each practice offers just 3 defensible variants and you exploit four of the seven: 3⁴ = 81 implicit specifications. At a 5% false-positive rate per spec, the probability at least one clears significance exceeds 98% under a true null — and that’s before any parameter grid multiplies it further. Flexibility compounds like leverage.
Tells a reader can spot in the published paper
- Fragile-boundary results: significance that lives at p = 0.049, or an effect present at the 30% trim but not 25%.
- Asymmetric robustness sections: ten checks that all pass, none that bind — real robustness sections contain results that fail, because honest grids have edges.
- Precision without preregistration: “we filter stocks below $4.83” — oddly specific thresholds are usually optimized ones.
- The moving outcome: abstract claims returns; tables deliver risk-adjusted-alpha-on-a-subsample.
- Distribution of reported t-stats: across the literature, t-values bunch suspiciously just above significance thresholds — a field-level fingerprint documented by the replication studies.
The fixes, by role
As a researcher: pre-register the specification in your experiment log before touching data; treat every post-hoc change as a new trial in your DSR count; report the full grid, including failures; and adopt the evaluate-once holdout as constitutional law.
As a reader: apply the t ≈ 3 hurdle for anything from a searched space; weight post-publication performance over in-sample tables; and prefer papers whose robustness sections have casualties.
Questions to ask when reading
- Which choices in this paper could have been made after seeing results — and does anything indicate they weren’t?
- Is the headline specification the one a naive researcher would have run first? If not, why not?
- What does the paper’s own appendix reveal about the size of the searched space?
- Do any reported checks actually threaten the result?
What our scoring captures — and misses
Our rigor axis partially proxies for p-hacking resistance: papers scoring high tend to have disclosed grids, symmetric robustness sections, and out-of-sample discipline — the practices that make hacking hard. But p-hacking’s essence is invisible flexibility, and no text-level score sees the specifications that were run and discarded. The score tells you the paper’s visible process is sound; the seven-practices lens is how you price the invisible one.
The mechanics: Backtest overfitting → · The arithmetic: statistical vs economic significance → · The field-level view: factor-zoo papers → · From the corpus: ARDL models: appropriate use vs p-hacking →