mlsanityData to 14 Oct 2026

Strong inductive biases provably prevent harmless interpolation

arXiv 2301.07605 ↗First posted 18 Jan 2023stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Classical wisdom suggests that estimators should avoid fitting noise to achieve good generalization. In contrast, modern overparameterized models can yield small test error despite interpolating noise -- a phenomenon often called "benign overfitting" or "harmless interpolation". This paper argues that the degree to which interpolation is harmless hinges upon the strength of an estimator's inductive bias, i.e., how heavily the estimator favors solutions with a certain structure: while strong inductive biases prevent harmless interpolation, weak inductive biases can even require fitting noise to generalize well. Our main theoretical result establishes tight non-asymptotic bounds for high-dimensional kernel regression that reflect this phenomenon for convolutional kernels, where the filter size regulates the strength of the inductive bias. We further provide empirical evidence of the same behavior for deep neural networks with varying filter sizes and rotational invariance.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

34 of 50 references.

References · 34

  1. 2016Wide Residual Networkscited by 3,070
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2020Overfitting in adversarially robust deep learningcited by 506
  6. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  7. 2019Linearized two-layers neural networks in high dimensioncited by 155
  8. 2020Benign overfitting in ridge regressioncited by 153
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  12. 2018Does data interpolation contradict statistical optimality?cited by 95
22 more
  1. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 89
  2. 2021Learning with invariances in random features and kernel modelscited by 89
  3. 2021Augmentation Strategies for Learning with Noisy Labelscited by 86
  4. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  5. 2020How benign is benign overfitting?cited by 40
  6. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  7. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  8. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  9. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  10. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  11. 2017Early stopping for kernel boosting algorithms: A general analysis with localized complexitiescited by 27
  12. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  13. 2021Foolish Crowds Support Benign Overfittingcited by 24
  14. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 22
  15. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 22
  16. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  17. 2019Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustnesscited by 18
  18. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  19. 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
  20. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  21. 2020How Does a Neural Network's Architecture Impact Its Robustness to Noisy Labels?cited by 12
  22. 2020Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 1; 2026: 1.

10 papers.

  1. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  2. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 11
  3. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 11
  4. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  5. 2024Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisationcited by 9
  6. 2024High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularizationcited by 4
  7. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 2
  8. 2025On the Saturation Effects of Spectral Algorithms in Large Dimensionscited by 2
  9. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  10. 2026Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensionscited by 0