Paper: arXiv 2605.09712

Authors: Philippe Goulet Coulombe

Abstract

Average forecast accuracy is not the same as forecast reliability. I treat forecast loss differentials relative to a benchmark as a return series. I then evaluate these returns using risk-adjusted performance measures from finance, including the Sharpe ratio, Sortino ratio, Omega ratio, and drawdown-based metrics. I also introduce the Edge Ratio capturing a model’s propensity to deliver uniquely informative predictions relative to the forecasting frontier. I apply this framework to U.S. macroeconomic forecasting, comparing econometric benchmarks, machine learning models, a foundation model (TabPFN), and the Survey of Professional Forecasters. While it is often feasible to beat professional forecasters in terms of average accuracy, it is much harder to beat them on a risk-adjusted basis. They rarely exhibit catastrophic failures and often achieve high Edge Ratios, plausibly reflecting the value of contextual judgment. Nonetheless, selected machine learning methods deliver attractive risk profiles for specific targets. The framework naturally extends to meta-analyses across targets, horizons, and samples, illustrated with a density forecast evaluation and the M4 competition.

Complexity vs Empirical Score

  • Math Complexity: 4.5/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on the empirical application of finance metrics to forecast performance rather than new mathematical derivations, utilizing extensive backtesting via macroeconomic data, the M4 competition, and various machine learning models. Its strength lies in the robust, systematic evaluation of forecast reliability across diverse datasets rather than theoretical density or complex mathematical proofs.

Research Flowchart

  flowchart TD
    A[<b>Goal:</b> Quantify Risk-Return Tradeoff in Forecasting] --> B[<b>Data:</b> US Macro Forecasts, SPF, M4 Competition]
    B --> C[<b>Method:</b> Treat Loss Differentials as Financial Returns]
    C --> D[<b>Processes:</b> Sharpe, Sortino, Omega, and Edge Ratios]
    D --> E[<b>Findings:</b> Difficulty in beating SPF on risk-adjusted basis]
    E --> F[<b>Outcome:</b> ML models effective for niche targets; contextual judgment is valuable]

Cited References (61)

  1. LGB+: A Macroeconomic Forecasting Road Test (2026) — link
  2. ChatMacro: Evaluating Inflation Forecasts of Generative AI (2026) — link
  3. Dual Interpretation of Machine Learning Forecasts (2024) — link
  4. Macroeconomic Forecasting with Large Language Models (2024) — link
  5. From Reactive to Proactive Volatility Modeling with Hemisphere Neural Networks (2023) — link
  6. Surveying Generative AI’s Economic Expectations (2023) — link
  7. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022) — link
  8. A Neural Phillips Curve and a Deep Output Gap (2022) — link
and 53 more
  1. Random Forest (2021) — link
  2. Tail Forecasting with Multivariate Bayesian Additive Regression Trees (2021) — link
  3. To Bag is to Prune (2020) — link
  4. FRED-QD: A Quarterly Database for Macroeconomic Research (2020) — link
  5. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting (2020) — link
  6. Forecasting in the Presence of Instabilities: How We Know Whether Models Predict Well and How to Improve Them (2019) — link
  7. How is Machine Learning Useful for Macroeconomic Forecasting? (2019) — link
  8. Forecasting Inflation in a Data-Rich Environment: The Benefits of Machine Learning Methods (2019) — link
  9. The M4 Competition: Results, findings, conclusion and way forward (2018) — link
  10. LightGBM: A Highly Efficient Gradient Boosting Decision Tree (2017)
  11. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks (2017) — link
  12. Forecaster’s Dilemma: Extreme Events and Forecast Evaluation (2015) — link
  13. Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings (2015) — link
  14. Ancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC estimation of stochastic volatility models (2014) — link
  15. Inference with normal-gamma prior distributions in regression problems (2010) — link
  16. Making and Evaluating Point Forecasts (2009) — link
  17. Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters (2009) — link
  18. BART: Bayesian Additive Regression Trees (2008) — link
  19. Properties of Optimal Forecasts under Asymmetric Loss and Nonlinearity (2007) — link
  20. Do Macro Variables, Asset Markets or Surveys Forecast Inflation Better? (2007) — link
  21. Strictly Proper Scoring Rules, Prediction, and Estimation (2007) — link
  22. A Test for Superior Predictive Ability (2005) — link
  23. Maximum Drawdown (2005) — link
  24. Tests of Conditional Predictive Ability (2003) — link
  25. Forecasting Using Principal Components From a Large Number of Predictors (2002) — link
  26. A Reality Check for Data Snooping (2000) — link
  27. Outline of forecast theory using generalized cost functions (1999) — link
  28. Economic and Statistical Measures of Forecast Accuracy (1999) — link
  29. Optimal Prediction Under Asymmetric Loss (1997) — link
  30. Testing the equality of prediction mean squared errors (1997) — link
  31. Asymptotic Inference about Predictive Ability (1996) — link
  32. Comparing Predictive Accuracy (1995) — link
  33. Performance Measurement in a Downside Risk Framework (1994) — link
  34. Asymmetric Least Squares Estimation and Testing (1987) — link
  35. Regret Theory: An alternative theory of rational choice under uncertainty Review of Economic Studies (1982) — link
  36. The Theory of Statistical Decision (1951) — link
  37. AlphaPortfolio: Direct Construction Through Deep Reinforcement Learning and Interpretable AI (2021)
  38. The M4 Competition: 100,000 time series and 61 forecasting methods (2020) — link
  39. Sequential gradient boosting with histogram-based splits; learning rate, depth, and sampling fractions cross-validated with early stopping (2017)
  40. Autoregressive model with stochastic volatility. The log-variance follows an AR(1) process, estimated via MCMC using the algorithm of Kastner and Frühwirth-Schnatter (2014)
  41. Kernel Ridge Regression (2013) — link
  42. Forecasting Inflation (2011) — link
  43. Realistic evaluation of real-time forecasts in the Survey of Professional Forecasters (2010)
  44. The Model Confidence Set ∗ (2009)
  45. MUTUAL FUND PERFORMANCE* (2007) — link
  46. Drawdown Measure in Portfolio Optimization (2005) — link
  47. Downside Risk ∗ (2004) — link
  48. A Universal Performance Measure (2002)
  49. Augments the AR(4) with four principal components extracted from the predictor panel, exploiting cross-variable information while avoiding the curse of dimensionality (2002)
  50. Random Forests (2001) — link
  51. FORECAST EVALUATION — link
  52. Fourth-order autoregression on the target variable. The benchmark that any serious forecaster must beat
  53. Ridge Regression. High-dimensional linear prediction with L 2 penalty; regularization parameter λ selected by cross-validation

0 of 61 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.