mlsanityData to 14 Oct 2026

Beyond Benign Overfitting in Nadaraya-Watson Interpolators

arXiv 2502.07480 ↗First posted 11 Feb 2025cs.LG math.ST stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

In recent years, there has been much interest in understanding the generalization behavior of interpolating predictors, which overfit on noisy training data. Whereas standard analyses are concerned with whether a method is consistent or not, recent observations have shown that even inconsistent predictors can generalize well. In this work, we revisit the classic interpolating Nadaraya-Watson (NW) estimator (also known as Shepard's method), and study its generalization capabilities through this modern viewpoint. In particular, by varying a single bandwidth-like hyperparameter, we prove the existence of multiple overfitting behaviors, ranging non-monotonically from catastrophic, through benign, to tempered. Our results highlight how even classical interpolating methods can exhibit intricate generalization behaviors. In addition, for the purpose of tuning the hyperparameter, the results suggest that over-estimating the intrinsic dimension of the data is less harmful than under-estimating it. Numerical experiments complement our theory, demonstrating the same phenomena.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

35 of 53 references.

References · 35

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  5. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  6. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  7. 2019Two models of double descent for weak featurescited by 195
  8. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  9. 2018To understand deep learning we need to understand kernel learningcited by 187
  10. 2020Benign overfitting in ridge regressioncited by 153
  11. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  12. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
23 more
  1. 2018Does data interpolation contradict statistical optimality?cited by 95
  2. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  3. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  4. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  5. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  6. 2022The Implicit Bias of Benign Overfittingcited by 38
  7. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  8. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  9. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  10. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  11. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  12. 2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13
  13. 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
  14. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  15. 2022Wide and Deep Neural Networks Achieve Optimality for Classificationcited by 12
  16. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  17. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  18. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  19. 2024A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regressioncited by 7
  20. 2024Context-Scaling versus Task-Scaling in In-Context Learningcited by 6
  21. 2023Interpolation Learning With Minimum Description Lengthcited by 5
  22. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  23. 2024A generalized neural tangent kernel for surrogate gradient learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Harmful Overfitting in Sobolev Spacescited by 1
  2. 2026A mathematical theory of balancing relational generalization and memorizationcited by 0