Paper: arXiv 2601.22200

Authors: Luis Ontaneda Mijares, Nick Firoozye

Abstract

Overparameterized models have recently challenged conventional learning theory by exhibiting improved generalization beyond the interpolation limit, a phenomenon known as benign overfitting. This work introduces Adaptive Benign Overfitting (ABO), extending the recursive least-squares (RLS) framework to this regime through a numerically stable formulation based on orthogonal-triangular updates. A QR-based exponentially weighted RLS (QR-EWRLS) algorithm is introduced, combining random Fourier feature mappings with forgetting-factor regularization to enable online adaptation under non-stationary conditions. The orthogonal decomposition prevents the numerical divergence associated with covariance-form RLS while retaining adaptability to evolving data distributions. Experiments on nonlinear synthetic time series confirm that the proposed approach maintains bounded residuals and stable condition numbers while reproducing the double-descent behavior characteristic of overparameterized models. Applications to forecasting foreign exchange and electricity demand show that ABO is highly accurate (comparable to baseline kernel methods) while achieving speed improvements of between 20 and 40 percent. The results provide a unified view linking adaptive filtering, kernel approximation, and benign overfitting within a stable online learning framework.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper provides a strong theoretical and algorithmic bridge between benign overfitting, recursive least squares (RLS), and random Fourier features. It is well-written with solid mathematical foundation and promising empirical evaluation across synthetic, FX, and electricity demand datasets.

Research Flowchart

  flowchart TD
    A["Research Goal: Enable stable benign overfitting in online non-stationary time-series forecasting"] --> B["Inputs: Non-stationary Time Series<br/>(Synthetic, Forex, Electricity)"]
    B --> C["Methodology: Feature Mapping & Regularization<br/>(Random Fourier Features + Forgetting Factors)"]
    C --> D["Computational Process: QR-EWRLS Algorithm<br/>(Orthogonal-Triangular Updates for Stability)"]
    D --> E["Outcomes: Prevents Divergence & Reproduces Double-Descent"]
    D --> F["Outcomes: High Accuracy with 20-40% Speedup vs Kernel Baselines"]

Cited References (58)

  1. Generalized Inverses of Linear Transformations (2009) — link
  2. Representations for the Generalized Inverse of a Partitioned Matrix (1964)
  3. Matrix Computations (1996)
  4. Kernel Adaptive Filtering: A Comprehensive Introduction (2011)
  5. Basic Econometrics (2003)
  6. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2009)
  7. Accuracy and Stability of Numerical Algorithms (2002) — link
  8. Updating the QR Factorization and the Least Squares Problem (2008)
and 50 more
  1. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity (2005)
  2. Numerical Linear Algebra (1997)
  3. A Method for Recursive Least Squares Filtering Based upon an Inverse QR Decomposition (1993)
  4. Rank Factorization and Moore-Penrose Inversion (1976)
  5. Two Models of Double Descent for Weak Features (2020) — link
  6. Benign Overfitting in Linear Regression (2020) — link
  7. Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018) — link
  8. Neural Signature Kernels as Infinite-Width-Depth Limits of Controlled ResNets (2023) — link
  9. Universal Kernels (2006) — link
  10. On the Relation Between Universality, Characteristic Kernels and RKHS Embedding of Measures (2011) — link
  11. On Exact Computation with an Infinitely Wide Neural Net (2019) — link
  12. Random Features for Large-Scale Kernel Machines (2007) — link
  13. Weighted Sums of Random Kitchen Sinks: Replacing Minimization with Randomization in Learning (2008)
  14. Breaking the Curse of Dimensionality with Convex Neural Networks (2017)
  15. Approximation by Superpositions of a Sigmoidal Function (1989)
  16. Approximation Capabilities of Multilayer Feedforward Networks (1991)
  17. Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2022) — link
  18. The Generalization Error of Random Features Regression: Precise Asymptotics and Double Descent Curve (2022)
  19. Benign overfitting in ridge regression (2023)
  20. Benign Overfitting and Noisy Features (2023)
  21. RLS Algorithm With Convex Regularization (2011) — link
  22. Zero-Attracting Recursive Least Squares Algorithms (2017) — link
  23. Adaptive projected subgradient method and its acceleration techniques (2004) — link
  24. Zero-Attracting Recursive Least Squares Algorithms (2017) — link
  25. Feature Weighting and Double Descent in High-Frequency Interpolation (2023) — link
  26. Empirical Properties of Asset Returns: Stylized Facts and Statistical Issues (2001)
  27. On short-time Fourier transform with single-sided exponential window (1996) — link
  28. Handbook of Financial Time Series (2009)
  29. Time-Frequency/Time-Scale Analysis (1998)
  30. Fundamentals of Adaptive Filtering (2003)
  31. Adaptive Filters (2008)
  32. Adaptive Filter Theory (2002)
  33. QRD–RLS Adaptive Filtering (2009)
  34. Forecasting with Exponential Smoothing: The State Space Approach (2008)
  35. Time Series Analysis by State Space Methods (2012)
  36. The Kernel Least-Mean-Square Algorithm (2008)
  37. Linear Estimation (2000)
  38. Nonlinear Time Series: Analysis, Methods and Applications (2004)
  39. Two models of double descent for weak features (2019)
  40. Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019)
  41. Benign overfitting in linear regression (2020)
  42. Efficient recursive least squares solver for rank-deficient matrices (2021) — link
  43. Econometric applications of positive rank‐one modifications of the symmetric factorization of a positive semi‐definite matrix (1999) — link
  44. Understanding Deep Learning Requires Rethinking Generalization (2017) — link
  45. On the Approximation Properties of Random ReLU Features (2018) — link
  46. ElectricityLoadDiagrams20112014 (2015)
  47. Least squares modifications with inverse factorizations: Parallel implications (1989) — link
  48. A Sliding-Window Kernel RLS Algorithm and Its Application to Nonlinear Channel Identification (2006) — link
  49. Optuna: A Next-generation Hyperparameter Optimization Framework (2019)
  50. On short-time Fourier transform with single-sided exponential window (1996) — link

0 of 58 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.