mlsanityData to 14 Oct 2026

Foolish Crowds Support Benign Overfitting

arXiv 2110.02914 ↗First posted 6 Oct 2021stat.ML cs.LG math.STcited by 24

Not on the map. Why?

Abstract (arXiv)

We prove a lower bound on the excess risk of sparse interpolating procedures for linear regression with Gaussian data in the overparameterized regime. We apply this result to obtain a lower bound for basis pursuit (the minimum ℓ1-norm interpolant) that implies that its excess risk can converge at an exponentially slower rate than OLS (the minimum ℓ2-norm interpolant), even when the ground truth is sparse. Our analysis exposes the benefit of an effect analogous to the "wisdom of the crowd", except here the harm arising from fitting the textitnoise is ameliorated by spreading it among many directions -- the variance reduction arises from a textitfoolish crowd.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 11 of its 24 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

11 of 24 references.

References · 11

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  4. 2020Benign overfitting in ridge regressioncited by 153
  5. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  6. 2019Harmless interpolation of noisy data in regressioncited by 121
  7. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  8. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  9. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  10. 2020On the robustness of minimum norm interpolators and regularized empirical risk minimizerscited by 19
  11. 2020The Geometry of Uniqueness, Sparsity and Clustering in Penalized Estimationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 5; 2024: 3; 2025: 3; 2026: 3.

24 papers.

Sort:

By area:

  1. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  2. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  3. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 22
  4. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  5. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  6. 2021Benign Overfitting in Adversarially Robust Linear Classificationcited by 11
  7. 2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11
  8. 2023Strong inductive biases provably prevent harmless interpolationcited by 10
  9. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  10. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  11. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  12. 2022A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictorscited by 7
12 more
  1. 2023Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?cited by 7
  2. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 6
  3. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  4. 2025Scanning Trojaned Models Using Out-of-Distribution Samplescited by 3
  5. 2025Transfer Learning for Benign Overfitting in High-Dimensional Linear Regressioncited by 3
  6. 2022Surprises in adversarially-trained linear regressioncited by 2
  7. 2025Optimal Implicit Bias in Linear Regressioncited by 1
  8. 2026Harmful Overfitting in Sobolev Spacescited by 1
  9. 2026De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Predictioncited by 1
  10. 2023Bayesian Analysis for Over-parameterized Linear Model via Effective Spectracited by 0
  11. 2024Benign Overfitting under Learning Rate Conditions for α Sub-exponential Inputcited by 0
  12. 2026Benign Overfitting for General Norms and Distributionscited by 0