Combinatorially Symmetric Cross-Validation (CSCV) Explained

Definition. Combinatorially Symmetric Cross-Validation (CSCV), from Bailey, Borwein, López de Prado and Zhu’s work on backtest overfitting, estimates the Probability of Backtest Overfitting (PBO): the probability that the strategy configuration you selected because it won in-sample would rank in the bottom half out-of-sample. Where the deflated Sharpe ratio benchmarks your best result against luck’s expected maximum, CSCV directly measures how unstable your selection is — no distributional assumptions about returns required, only your own trial matrix.

The construction

Start from what a search process actually produces: a matrix M of returns with T rows (time) and N columns (every configuration you evaluated — the honest N, not the flattering one).

1. Partition the T rows into S equal, contiguous blocks (S = 16 is a common choice), preserving temporal order within blocks.
2. Form all combinations of S/2 blocks as the in-sample set; the complementary S/2 blocks are out-of-sample. That’s C(16, 8) = 12,870 symmetric splits — every split has a mirror, so in- and out-samples are statistically interchangeable, unlike a single chronological split.
3. For each split: rank all N configurations in-sample; take the in-sample winner n*; find its out-of-sample rank among all N; convert to a relative rank ω ∈ (0,1).
4. PBO = the fraction of splits where ω < 0.5 — the in-sample winner landed in the OOS bottom half. The logit λ = ln(ω/(1−ω)) across splits gives a full distribution, not just the headline probability.

from itertools import combinations
import numpy as np

def pbo(M, S=16):                      # M: T x N matrix of per-period returns
    blocks = np.array_split(np.arange(len(M)), S)
    below = 0; splits = list(combinations(range(S), S // 2))
    for c in splits:
        ins  = np.concatenate([blocks[i] for i in c])
        outs = np.concatenate([blocks[i] for i in range(S) if i not in c])
        sr = lambda X: X.mean(0) / (X.std(0) + 1e-12)
        winner = sr(M[ins]).argmax()
        w = (sr(M[outs]) < sr(M[outs])[winner]).mean()   # OOS relative rank of winner
        below += (w < 0.5)
    return below / len(splits)

Run it on the thousand-coin-flips demo and PBO comes out near 0.5 — the in-sample “winner” is a coin toss out-of-sample, exactly what overfitting means. A selection process with real signal pushes PBO toward 0.

Reading the number

• PBO ≲ 0.2: selection is finding something persistent; proceed to the rest of the gauntlet.
• PBO ≈ 0.5: your selection is noise — the best backtest carries no information about OOS ranking.
• PBO > 0.5: worse than noise; the search is anti-selecting (usually a tell of strong mean-reverting noise fit).

Report it alongside the winner’s Sharpe: “SR 1.8, PBO 0.11, N = 240 configurations” is a claim; “SR 1.8” is a mood.

Honest limitations

• It needs the trial matrix. CSCV audits the configurations you kept records for — untracked exploratory trials escape it, which is the same reason the experiment log is the foundational defense.
• Block boundaries leak slightly with overlapping features/labels; apply the same purge-and-embargo hygiene at block edges.
• Combinations aren’t causal time: OOS blocks can predate IS blocks, so CSCV measures selection stability, not deployment realism — it complements walk-forward, never replaces it.
• It says nothing about data biases: a universe with survivorship or a pipeline with leakage passes CSCV proudly and is still fiction.

Questions to ask when a paper reports (or should report) PBO

1. Does N include the full search grid, or only the finalists?
2. Is the performance metric in step 3 the same one the paper’s conclusions use?
3. With S blocks, is each block long enough to contain a regime, or is the split shredding regime structure into confetti?
4. If the paper reports no overfitting diagnostics at all, what does its search-space size imply the PBO plausibly is?

What our scoring captures — and misses

Papers that report PBO, DSR, or disclosed trial matrices earn top marks on the rigor axis — they’re also rare, which itself is informative about the field. Our score can’t compute PBO for a paper (that requires the trial matrix only authors possess), so the score rewards the disclosure, and the reproduction process is where you generate the matrix yourself for results that matter.

Reference: Bailey, Borwein, López de Prado & Zhu, “The Probability of Backtest Overfitting” (Journal of Computational Finance).

The family: Backtest overfitting → · Deflated Sharpe → · Walk-forward →