mlsanityData to 7 Oct 2026

Interpolation can hurt robust generalization even when there is no noise

arXiv 2108.02883 ↗First posted 5 Aug 2021stat.ML cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Numerous recent works show that overparameterization implicitly reduces variance for min-norm interpolators and max-margin classifiers. These findings suggest that ridge regularization has vanishing benefits in high dimensions. We challenge this narrative by showing that, even in the absence of noise, avoiding interpolation through ridge regularization can significantly improve generalization. We prove this phenomenon for the robust risk of both linear regression and classification and hence provide the first theoretical result on robust overfitting.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 38 of its 61 references that are in the map. First posted on arXiv 5 Aug 2021.

Builds on

38 of 61 references.

References · 38

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2020Overfitting in adversarially robust deep learningcited by 504
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  10. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
26 more
  1. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  2. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  3. 2018To understand deep learning we need to understand kernel learningcited by 186
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2019Linearized two-layers neural networks in high dimensioncited by 155
  6. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 148
  7. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  8. 2018Invariance, Causality and Robustnesscited by 123
  9. 2019Harmless interpolation of noisy data in regressioncited by 121
  10. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  11. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  12. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  13. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  14. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 79
  15. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  16. 2020Precise Tradeoffs in Adversarial Training for Linear Regressioncited by 76
  17. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  18. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  19. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  20. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  21. 2020Precise Statistical Analysis of Classification Accuracies for Adversarial Trainingcited by 49
  22. 2020How benign is benign overfitting?cited by 40
  23. 2019The Impact of Regularization on High-dimensional Logistic Regressioncited by 35
  24. 2020Domain adaptation under structural causal modelscited by 28
  25. 2021Provable Robustness of Adversarial Training for Learning Halfspaces with Noisecited by 10
  26. 2021Binary classification with corrupted labelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 4; 2024: 1; 2025: 3; 2026: 2.

15 papers.

Sort:

  1. 2022Hierarchical Shrinkage: improving the accuracy and interpretability of tree-based methodscited by 18
  2. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  3. 2022Why adversarial training can hurt robust accuracycited by 12
  4. 2023Strong inductive biases provably prevent harmless interpolationcited by 9
  5. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 8
  6. 2024The Surprising Harmfulness of Benign Overfitting for Adversarial Robustnesscited by 7
  7. 2022Margin-based sampling in high dimensions: When being active is less efficient than staying passivecited by 3
  8. 2022Interpolation and Regularization for Causal Learningcited by 2
  9. 2023On Achieving Optimal Adversarial Test Errorcited by 1
  10. 2025Towards Better Generalization via Distributional Input Projection Networkcited by 1
  11. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 1
  12. 2026On damage of interpolation to adversarial robustness in regressioncited by 1
3 more
  1. 2022Rethinking Cost-sensitive Classification in Deep Learning via Adversarial Data Augmentationcited by 0
  2. 2025A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularizationcited by 0
  3. 2026Adversarial Robustness of NTK Neural Networkscited by 0