How to Interpret Confidence Intervals in Strategy Research

Definition. A 95% confidence interval around a strategy statistic says: under this procedure, intervals built this way cover the true value 95% of the time. It is a statement about the estimator’s noise, not a probability statement about your strategy — and in strategy research its two chronic problems are that honest intervals are shockingly wide, and published ones are systematically too narrow, because selection happened before the interval was drawn.

The numbers nobody internalizes

The standard error of an annualized Sharpe estimated from T years of (roughly i.i.d.) daily data is approximately √((1 + SR²/2)/T) — for practical purposes, near 1/√T for moderate Sharpes. Turn that into intervals:

Read the second row again: after four years, “SR = 1” is statistically indistinguishable from zero at conventional confidence. Consequences: comparing two strategies with |ΔSR| < ~0.5 on a few years of data is reading noise; manager evaluation on 3-year windows is astrology with spreadsheets; and any paper drawing conclusions from second-decimal Sharpe differences has confused precision with information. Non-normality widens everything further — negative skew and fat tails make the effective interval wider than the Gaussian formula admits.

Computing intervals honestly: the block bootstrap

Market returns are autocorrelated and volatility-clustered, so the i.i.d. bootstrap understates uncertainty. The standing tool is the block bootstrap (or stationary bootstrap with geometric block lengths — the generator menu):

def block_ci(rets, stat, n_boot=2000, block=21):
    T = len(rets); n_blocks = T // block + 1
    out = []
    for _ in range(n_boot):
        starts = rng.integers(0, T - block, n_blocks)
        sample = np.concatenate([rets[s:s+block] for s in starts])[:T]
        out.append(stat(sample))
    return np.percentile(out, [2.5, 97.5])

Block length should cover the dependence horizon (weeks for daily equity returns; longer for regime-sensitive strategies). The same machinery gives intervals on drawdowns, quantiles, and anything else without a closed form. For simulation outputs, remember the estimate itself has sampling error worth reporting.

The selection problem: why published intervals lie

A confidence interval is valid for a pre-specified statistic. The interval around your best-of-N backtest is a different object entirely: the selection already consumed the luck the interval thinks is still available, so naive CIs on selected results are too narrow — often absurdly so. The corrections live in the deflated Sharpe / CSCV family; the reading rule is simpler: an interval is only as honest as the trial count behind it, and papers reporting tight intervals on searched results have reported the wrong distribution.

The checklist

• ☐ Interval computed with dependence-aware methods (block bootstrap / HAC errors), not i.i.d. formulas
• ☐ Width sanity-checked against the 1/√T table — suspiciously tight intervals mean a method error
• ☐ Statistic pre-specified, or interval adjusted for selection
• ☐ Skew/kurtosis reported alongside (the interval’s shape matters, not just width)
• ☐ Comparisons between strategies use the interval on the difference, with common time periods

Questions to ask when reading a paper

1. Is any interval or standard error reported at all — and if not, what would the 1/√T arithmetic say about the headline number?
2. What dependence structure did the errors assume, and does the strategy’s holding period violate it?
3. Was this statistic selected before or after seeing results?
4. Do the paper’s conclusions survive at the interval’s unfavorable edge?

What our scoring captures — and misses

Reporting uncertainty at all is rare enough that it moves papers up the rigor axis — intervals, standard errors, and bootstrap sections are among the strongest textual signals of care we score on. The miss is the selection problem: a beautifully computed interval on a silently selected result scores well and misleads anyway. Pair the score with the trial-count question, always.

The width arithmetic’s sequel: Deflated Sharpe → · The dependence machinery: Auditing a Monte Carlo → · The comparison discipline: significance guide →