Implementation Risk in Portfolio Backtesting: A Previously Unquantified Source of Error

Paper: arXiv 2603.20319

Authors: Dong Yin, Takeshi Miki, Vladislav Lesnichenko, Vasyl Gural

Abstract

Portfolio backtesting is the primary tool for evaluating investment strategies before deployment, yet practitioners implicitly assume that different engines produce identical results for the same strategy. we formalise implementation risk, the systematic divergence in backtested portfolio metrics arising solely from differences in how engines implement the same logical strategy, and propose four metrics grounded in metrology to quantify it: engine sensitivity, implementation uncertainty interval, divergence amplification factor, and conclusion stability index. we execute 15 benchmark strategies through five independent open-source engines on 30 non-overlapping stratified asset buckets comprising 180 s&p 500 stocks under four transaction-cost regimes. at zero cost, all five engines agree exactly (maximum divergence 0.000%), isolating transaction-cost implementation as the sole source of disagreement. under nonzero costs, divergence is structured and predictable (spearman rho = 0.93 with cost intensity), remaining below 0.75 percentage points for most strategies but reaching 3.71% for high-turnover rotation strategies. source-code forensics uncovered seven previously undocumented defects across three engines, abstracted into a five-category failure-mode taxonomy. all engines agree on the sign of every performance metric (conclusion stability index = 1), so implementation risk does not alter investment decisions for the strategies studied but introduces measurable ambiguity in performance attribution. code and benchmark data are publicly available.

Complexity vs Empirical Score

• Math Complexity: 4.5/10
• Empirical Rigor: 8.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper introduces a highly novel metrological framework to quantify implementation risk across backtesting engines, supported by rigorous benchmarking across 15 strategies and 5 engines. The mathematical burden is moderate, focusing on empirical quantification and source-code forensics rather than complex theoretical proofs.

Research Flowchart

  flowchart TD
    A["<b>Research Goal:</b> Formalize & quantify implementation risk in portfolio backtesting"] --> B["<b>Data & Inputs:</b> 15 strategies across 180 S&P 500 stocks (30 buckets) & 4 cost regimes"]
    B --> C["<b>Execution & Forensics:</b> Run strategies across 5 open-source engines & perform source-code forensics"]
    C --> D["<b>Metrology Framework:</b> Evaluate engines using 4 metrics (Sensitivity, Uncertainty, Amplification, CSI)"]
    D --> E["<b>Key Finding 1:</b> Transaction costs are sole source of error (0.00% div. at zero cost; up to 3.71% at non-zero)"]
    D --> F["<b>Key Finding 2:</b> Identified 7 defects across engines, leading to a 5-category failure-mode taxonomy"]
    D --> G["<b>Key Finding 3:</b> Conclusion Stability Index = 1 (decision signs hold, but attribution ambiguity exists)"]

Cited References (31)

1. Supervisory Guidance on Model Risk Management (2011)
2. Computational Reproducibility in Finance: Evidence from 1,000 Tests (2024) — link
3. Measuring the model risk-adjusted performance of machine learning algorithms in credit default prediction (2022) — link
4. SAFE artificial intelligence in finance (2023)
5. The Deflated S harpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality (2014)
6. Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (2014)
7. A Reality Check for Data Snooping (2000)
8. A Test for Superior Predictive Ability (2005)

0 of 31 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.