Paper: arXiv 2610.40366

Authors: Raphael Coelho

Abstract

When two forecasts of a tail, such as prediction intervals at high coverage, extreme quantiles or expected shortfall (ES), are compared by a consistent score, the score difference is known in advance except on the days on which the outcome falls beyond a forecast. The number of these exceedances, not the number of observations, is the effective sample size. For intervals at miscoverage level $α$, the number of observations a comparison needs grows like $1/α$ while the number of exceedances it needs does not, and for some pairs of forecasts no test can do better. A confidence sequence that is valid at all times and needs no distributional assumption attains this rate; Hoeffding-type procedures, calibrated by the range of the score difference, need $1/α$ times more. When exceedances are scarce, the Diebold-Mariano test declares the less conservative forecaster better with at least, asymptotically, the probability of observing no exceedance, whichever forecaster is actually better; the usual small-sample corrections do not remove this, while a bootstrap-t version largely does. For (value-at-risk, ES) forecasts, a less conservative ES can be shown without assumptions to be worse on the days on which it is less conservative, but never to be better without an assumption on the loss tail. In 250-day windows of 97.5% forecasts for seven stock indices, 18.4% of significant Diebold-Mariano results favouring the less conservative model contradict the ranking on the remaining days, against 6.9% of those favouring the more conservative one.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a highly mathematical and rigorous analysis of tail forecast comparison, introducing novel insights into effective sample size and Diebold-Mariano test limitations. The empirical sections, while present, could benefit from more explicit data and code availability for full reproducibility, but the methodology is well-detailed.

Research Flowchart

  flowchart TD
    A[Research Goal: Compare Tail Forecasts & Assess Sample Size Impact] --> B{Methodology: Consistent Score Comparison & Effective Sample Size}
    B --> C[Data/Inputs: Tail Forecasts (Prediction Intervals, Extreme Quantiles, ES)]
    C --> D{Computational Processes: Confidence Sequences, Hoeffding-type Procedures, Diebold-Mariano Test, Bootstrap-t}
    D --> E[Empirical Analysis: 250-day windows, 97.5% forecasts, 7 Stock Indices]
    E --> F[Key Finding 1: Effective Sample Size = Number of Exceedances (not observations)]
    F --> G[Key Finding 2: Diebold-Mariano test issues with scarce exceedances; Bootstrap-t helps]
    G --> H[Key Finding 3: Less conservative ES can be worse without loss tail assumption]
    H --> I[Key Finding 4: DM results favouring less conservative model often contradict other days' ranking (18.4% vs 6.9%)]