Statistical Inference for Score Decompositions

Paper: arXiv 2603.04275

Authors: Timo Dimitriadis, Marius Puke

Abstract

We introduce inference methods for score decompositions, which partition scoring functions for predictive assessment into three interpretable components: miscalibration, discrimination, and uncertainty. Our estimation and inference relies on a linear recalibration of the forecasts, which is applicable to general multi-step ahead point forecasts such as means and quantiles due to its validity for both smooth and non-smooth scoring functions. This approach ensures desirable finite-sample properties, enables asymptotic inference, and establishes a direct connection to the classical Mincer-Zarnowitz regression. The resulting inference framework facilitates tests for equal forecast calibration or discrimination, which yield three key advantages. They enhance the information content of predictive ability tests by decomposing scores, deliver higher statistical power in certain scenarios, and formally connect scoring-function-based evaluation to traditional calibration tests, such as financial backtests. Applications demonstrate the method’s utility. We find that for survey inflation forecasts, discrimination abilities can differ significantly even when overall predictive ability does not. In an application to financial risk models, our tests provide deeper insights into the calibration and information content of volatility and Value-at-Risk forecasts. By disentangling forecast accuracy from backtest performance, the method exposes critical shortcomings in current banking regulation.

Complexity vs Empirical Score

• Math Complexity: 8.0/10
• Empirical Rigor: 8.5/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper addresses a critical gap in forecast evaluation by developing formal asymptotic inference for components of score decompositions (miscalibration and discrimination) across smooth and non-smooth loss functions. It combines rigorous theoretical econometrics with high practical utility, demonstrating how current score-based vs. calibration-based backtests can lead to contradictory conclusions in financial risk management and macro forecasting.

Research Flowchart

  flowchart TD
    A[Goal: Inferential Methods for Score Decompositions<br>Miscalibration, Discrimination, Uncertainty] --> B[Input: Multi-step Point Forecasts & Outcomes<br>Means and Quantiles]
    B --> C[Method: Linear Recalibration via<br>Mincer-Zarnowitz Regressions]
    C --> D[Compute: Score Decompositions & Hypothesis Tests<br>For Equal Calibration / Discrimination]
    D --> E[Application 1: Survey Inflation Forecasts]
    D --> F[Application 2: Financial Risk Models<br>Volatility & Value-at-Risk]
    E & F --> G[Outcomes: Disentangles Accuracy vs Backtesting,<br>Boosts Power, Exposes Regulatory Shortcomings]

Cited References (270)

1. HAC Covariance Matrix Estimation in Quantile Regression (2024) — link
2. Value-at-Risk Prediction: A Comparison of Alternative Strategies (2005) — link
3. Autoregressive Conditional Duration: A New Model for Irregularly Spaced Transaction Data (1998) — link
4. Backtesting Value-at-Risk: A Duration-Based Approach (2004) — link
5. Backtesting Value-at-Risk: A GMM Duration-Based Test (2010) — link
6. The Geometric-VaR Backtesting Method (2015) — link
7. Frontiers in VaR forecasting and backtesting (2016) — link
8. A new set of improved Value-at-Risk backtests (2014) — link

0 of 270 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.