mlsanityData to 14 Oct 2026

Benign Overfitting for General Norms and Distributions

arXiv 2609.33675 ↗First posted 27 Sep 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Understanding why predictors can generalize despite interpolating noisy training data is a central puzzle in machine learning. Most work on such "benign overfitting" studies minimum-2-norm linear regression, reflecting the inductive bias of gradient descent. However, modern optimizers such as Adam and Muon use non-Euclidean update geometries, favoring solutions associated with other norms. Analyzing regression for non-Euclidean norms is substantially more difficult, with known results essentially limited to Gaussians. In this paper, we develop a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions. As a special case, we prove that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions. Perhaps surprisingly, for the 1-norm, benign overfitting does not hold in general for well-behaved (but non-Gaussian) distributions, showing that existing positive 1-norm results rely crucially on Gaussianity. Our proof analyzes the geometry of the dual optimization problem, using concentration and central limit tools to show it is approximately Euclidean in many high-dimensional cases.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

30 of 43 references.

References · 30

  1. 2019Benign Overfitting in Linear Regressioncited by 549
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2019Two models of double descent for weak featurescited by 195
  6. 2024Old Optimizer, New Norm: An Anthologycited by 193
  7. 2020Benign overfitting in ridge regressioncited by 153
  8. 2019Harmless interpolation of noisy data in regressioncited by 121
  9. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  10. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  11. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  12. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
18 more
  1. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  2. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 53
  3. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  4. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  5. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  6. 2022The Implicit Bias of Benign Overfittingcited by 38
  7. 2024The Implicit Bias of Adam on Separable Datacited by 36
  8. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  9. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  10. 2021Foolish Crowds Support Benign Overfittingcited by 24
  11. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 22
  12. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  13. 2020On the robustness of minimum norm interpolators and regularized empirical risk minimizerscited by 19
  14. 2024Minimum-Norm Interpolation Under Covariate Shiftcited by 14
  15. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 11
  16. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  17. 2024A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regressioncited by 7
  18. 2026Minimum Norm Interpolation via the Local Theory of Banach Spaces: The Role of 2-Uniform Convexitycited by 2

Built on by

Not cited yet.