# Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Bias arXiv: https://arxiv.org/abs/2203.03597 · HTML: https://mlsanity.com/paper/2203.03597/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 55 references that are in the map. - First posted on arXiv 7 Mar 2022. ## Builds on (most cited first, 25 of 25) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017) - [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019) - [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019) - [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019) - [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014) - [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019) - [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020) - [Finite Versus Infinite Neural Networks: an Empirical Study](https://mlsanity.com/paper/2007.15801/) (2020) - [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020) - [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019) - [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015) - [Harmless interpolation of noisy data in regression](https://mlsanity.com/paper/1903.09139/) (2019) - [Classification vs regression in overparameterized regimes: Does the loss function matter?](https://mlsanity.com/paper/2005.08054/) (2020) - [A Model of Double Descent for High-dimensional Binary Linear Classification](https://mlsanity.com/paper/1911.05822/) (2019) - [Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting](https://mlsanity.com/paper/2106.09276/) (2021) - [Learning without Concentration](https://mlsanity.com/paper/1401.0304/) (2014) - [On Uniform Convergence and Low-Norm Interpolation Learning](https://mlsanity.com/paper/2006.05942/) (2020) - [Foolish Crowds Support Benign Overfitting](https://mlsanity.com/paper/2110.02914/) (2021) - [Learning with convolution and pooling operations in kernel methods](https://mlsanity.com/paper/2111.08308/) (2021) - [High-dimensional classification by sparse logistic regression](https://mlsanity.com/paper/1706.08344/) (2017) - [On the robustness of minimum norm interpolators and regularized empirical risk minimizers](https://mlsanity.com/paper/2012.00807/) (2020) - [Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression](https://mlsanity.com/paper/2112.04470/) (2021) - [Tight bounds for minimum l1-norm interpolation of noisy data](https://mlsanity.com/paper/2111.05987/) (2021) - [AdaBoost and robust one-bit compressed sensing](https://mlsanity.com/paper/2105.02083/) (2021) ## Built on by (most cited first, 22 of 22) - [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022) - [Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data](https://mlsanity.com/paper/2502.04664/) (2025) - [Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization](https://mlsanity.com/paper/2303.01462/) (2023) - [Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficiently](https://mlsanity.com/paper/2205.12808/) (2022) - [Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension](https://mlsanity.com/paper/2305.14077/) (2023) - [A Unified Approach to Controlling Implicit Regularization via Mirror Descent](https://mlsanity.com/paper/2306.13853/) (2023) - [Implicit Optimization Bias of Next-Token Prediction in Linear Models](https://mlsanity.com/paper/2402.18551/) (2024) - [A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/2210.12082/) (2022) - [Noisy Interpolation Learning with Shallow Univariate ReLU Networks](https://mlsanity.com/paper/2307.15396/) (2023) - [Strong inductive biases provably prevent harmless interpolation](https://mlsanity.com/paper/2301.07605/) (2023) - [A law of adversarial risk, interpolation, and label noise](https://mlsanity.com/paper/2207.03933/) (2022) - [Deep Linear Networks can Benignly Overfit when Shallow Ones Do](https://mlsanity.com/paper/2209.09315/) (2022) - [A Survey of Inductive Reasoning for Large Language Models](https://mlsanity.com/paper/2510.10182/) (2025) - [Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality](https://mlsanity.com/paper/2409.03891/) (2024) - [An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression](https://mlsanity.com/paper/2306.13185/) (2023) - [Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression](https://mlsanity.com/paper/2510.15337/) (2025) - [Minimum Norm Interpolation via the Local Theory of Banach Spaces: The Role of 2-Uniform Convexity](https://mlsanity.com/paper/2603.28956/) (2026) - [An Adaptive Tangent Feature Perspective of Neural Networks](https://mlsanity.com/paper/2308.15478/) (2023) - [Optimal Implicit Bias in Linear Regression](https://mlsanity.com/paper/2506.17187/) (2025) - [Closed-form ℓr norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓp bias](https://mlsanity.com/paper/2509.21181/) (2025) - [Tight bounds for maximum ℓ1-margin classifiers](https://mlsanity.com/paper/2212.03783/) (2022) - [Benign Overfitting for General Norms and Distributions](https://mlsanity.com/paper/2609.33675/) (2026) ## Abstract (arXiv) Good generalization performance on high-dimensional data crucially hinges on a simple structure of the ground truth and a corresponding strong inductive bias of the estimator. Even though this intuition is valid for regularized models, in this paper we caution against a strong inductive bias for interpolation in the presence of noise: While a stronger inductive bias encourages a simpler structure that is more aligned with the ground truth, it also increases the detrimental effect of noise. Specifically, for both linear regression and classification with a sparse ground truth, we prove that minimum ℓp-norm and maximum ℓp-margin interpolators achieve fast polynomial rates close to order 1/n for p > 1 compared to a logarithmic rate for p = 1. Finally, we provide preliminary experimental evidence that this trade-off may also play a crucial role in understanding non-linear interpolating models used in practice.