Paper: arXiv 2512.00916

Authors: Sotirios D. Nikolopoulos

Abstract

Evaluating rare-event forecasts is challenging because standard metrics collapse as event prevalence declines. Measures such as F1-score, AUPRC, MCC, and accuracy induce degenerate thresholds – converging to zero or one – and their values become dominated by class imbalance rather than tail discrimination. We develop a family of rare-event-stable (RES) metrics whose optimal thresholds remain strictly interior as the event probability approaches zero, ensuring coherent decision rules under extreme rarity. Simulations spanning event probabilities from 0.01 down to one in a million show that RES metrics maintain stable thresholds, consistent model rankings, and near-complete prevalence invariance, whereas traditional metrics exhibit statistically significant threshold drift and structural collapse. A credit-default application confirms these results: RES metrics yield interpretable probability-of-default cutoffs (4-9%) and remain robust under subsampling, while classical metrics fail operationally. The RES framework provides a principled, prevalence-invariant basis for evaluating extreme-risk forecasts.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper addresses a fundamental flaw in standard evaluation metrics under extreme class imbalance by establishing decision-theoretic foundations for Rare-Event-Stable (RES) metrics. The theoretical derivations are backed by extensive simulations across extreme probability regimes and a realistic credit-default empirical application.

Research Flowchart

  flowchart TD
    A[Research Goal: Evaluate Rare-Event Forecasts Without Metric Collapse] --> B[Identify Problem: Standard Metrics F1, AUPRC Drift Under Rarity]
    B --> C[Develop Framework: Rare-Event-Stable RES Metrics]
    C --> D[Inputs: Synthetic Data p=0.01 to 10⁻⁶ & Credit Default Data]
    D --> E[Compute Optimal Thresholds, Metric Stability & Model Rankings]
    E --> F[Findings: RES Maintains Stable Cutoffs 4-9% & Prevalence Invariance]

Cited References (39)

  1. Currency crisis early warning systems: Why they should be dynamic (2014) — link
  2. Evaluating probability forecasts for GDP declines using alternative methodologies (2013) — link
  3. Strictly proper scoring rules, prediction, and estimation (2007) — link
  4. Comparing density forecasts using threshold-and quantile-weighted scoring rules (2011) — link
  5. Likelihood-based scoring rules for comparing density forecasts in tails (2011) — link
  6. The decrease in confidence with forecast extremity (2025) — link
  7. Forecasting banking crises with dynamic panel probit models (2018) — link
  8. SRISK : A conditional capital shortfall measure (2017) — link
and 31 more
  1. The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature (2011) — link
  2. SMOTE : Synthetic Minority Over-sampling Technique (2002) — link
  3. Learning from imbalanced data (2009) — link
  4. Logistic regression in rare events data (2001) — link
  5. LOF : Identifying density-based local outliers (2000) — link
  6. Deep learning for anomaly detection: A review (2020) — link
  7. Behavior of prediction performance metrics with rare events (2026) — link
  8. Index for rating diagnostic tests (1950)
  9. Measuring classifier performance: a coherent alternative to the area under the ROC curve (2009) — link
  10. Evaluating diagnostic tests: The area under the ROC curve and its relatives (2010)
  11. ROC Graphs: Notes and practical considerations (2006)
  12. Predicting US recessions with dynamic binary response models (2008) — link
  13. False dichotomy alert: Improving subjective-probability estimates vs. raising awareness of systemic risk (2023) — link
  14. On single point forecasts for fat-tailed variables (2022) — link
  15. Precision-Recall-Gain Curves: PR Analysis Done Right (2015)
  16. The precision–recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets (2015) — link
  17. The relationship between Precision–Recall and ROC curves (2006) — link
  18. A comprehensive survey on rare event prediction (2024) — link
  19. The M6 forecasting competition: Bridging the gap between forecasting and investment decisions (2025) — link
  20. An introduction to the bootstrap (1993)
  21. A Comprehensive Survey of Data Mining-Based Accounting-Fraud Detection Research (2010) — link
  22. Financial fraud detection through the application of machine learning techniques: a literature review (2024) — link
  23. Focal loss for dense object detection (2017) — link
  24. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation (2011)
  25. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020) — link
  26. Estimating probabilities of default for low default portfolios (2011) — link
  27. The foundations of cost-sensitive learning (2001)
  28. A unifying view on dataset shift in classification (2012) — link
  29. A note on using the F-measure for evaluating record linkage algorithms (2018) — link
  30. LightGBM : A highly efficient gradient boosting decision tree (2017)
  31. Modeling Extreme Events: Time-Varying Extreme Tail Shape (2024) — link

0 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.