mlsanityData to 7 Oct 2026

Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Bias

arXiv 2203.03597 ↗First posted 7 Mar 2022stat.ML cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Good generalization performance on high-dimensional data crucially hinges on a simple structure of the ground truth and a corresponding strong inductive bias of the estimator. Even though this intuition is valid for regularized models, in this paper we caution against a strong inductive bias for interpolation in the presence of noise: While a stronger inductive bias encourages a simpler structure that is more aligned with the ground truth, it also increases the detrimental effect of noise. Specifically, for both linear regression and classification with a sparse ground truth, we prove that minimum ℓp-norm and maximum ℓp-margin interpolators achieve fast polynomial rates close to order 1/n for p > 1 compared to a logarithmic rate for p = 1. Finally, we provide preliminary experimental evidence that this trade-off may also play a crucial role in understanding non-linear interpolating models used in practice.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

25 of 55 references.

References · 25

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  10. 2020Benign overfitting in ridge regressioncited by 152
  11. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  12. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
13 more
  1. 2019Harmless interpolation of noisy data in regressioncited by 121
  2. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  3. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  4. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  5. 2014Learning without Concentrationcited by 50
  6. 2020On Uniform Convergence and Low-Norm Interpolation Learningcited by 24
  7. 2021Foolish Crowds Support Benign Overfittingcited by 24
  8. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  9. 2017High-dimensional classification by sparse logistic regressioncited by 19
  10. 2020On the robustness of minimum norm interpolators and regularized empirical risk minimizerscited by 18
  11. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  12. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  13. 2021AdaBoost and robust one-bit compressed sensingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 7; 2024: 2; 2025: 5; 2026: 1.

21 papers.

Sort:

By area:

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  3. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  4. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  5. 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 25
  6. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  7. 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 13
  8. 2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11
  9. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  10. 2023Strong inductive biases provably prevent harmless interpolationcited by 9
  11. 2022A law of adversarial risk, interpolation, and label noisecited by 8
  12. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
9 more
  1. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  2. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  3. 2023An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regressioncited by 5
  4. 2025Transfer Learning for Benign Overfitting in High-Dimensional Linear Regressioncited by 2
  5. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  6. 2025Closed-form ℓr norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓp biascited by 1
  7. 2022Tight bounds for maximum ℓ1-margin classifierscited by 0
  8. 2025Optimal Implicit Bias in Linear Regressioncited by 0
  9. 2026Benign Overfitting for General Norms and Distributionscited by 0