# Benign Overfitting and Noisy Features

arXiv: https://arxiv.org/abs/2008.02901 · HTML: https://mlsanity.com/paper/2008.02901/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2020.

## Built on by (most cited first, 18 of 18)

- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Towards an Understanding of Benign Overfitting in Neural Networks](https://mlsanity.com/paper/2106.03212/) (2021)
- [Bias-variance decomposition of overparameterized regression with random linear features](https://mlsanity.com/paper/2203.05443/) (2022)
- [Understanding Deep Learning via Decision Boundary](https://mlsanity.com/paper/2206.01515/) (2022)
- [Unveiling Multiple Descents in Unsupervised Autoencoders](https://mlsanity.com/paper/2406.11703/) (2024)
- [The Geometry of Over-parameterized Regression and Adversarial Perturbations](https://mlsanity.com/paper/2103.14108/) (2021)
- [How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?](https://mlsanity.com/paper/2510.17526/) (2025)
- [Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression](https://mlsanity.com/paper/2202.06054/) (2022)
- [Scanning Trojaned Models Using Out-of-Distribution Samples](https://mlsanity.com/paper/2501.17151/) (2025)
- [Benign Overfitting with Quantum Kernels](https://mlsanity.com/paper/2503.17020/) (2025)
- [Robustness of Mixtures of Experts to Feature Noise](https://mlsanity.com/paper/2601.14792/) (2026)
- [Learning Non-Vacuous Generalization Bounds from Optimization](https://mlsanity.com/paper/2206.04359/) (2022)
- [Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression](https://mlsanity.com/paper/2404.04498/) (2024)
- [Implicit Regularization for Multi-label Feature Selection](https://mlsanity.com/paper/2411.11436/) (2024)
- [Copula-enhanced Vision Transformer for high myopia diagnosis through OU UWF fundus images](https://mlsanity.com/paper/2501.06540/) (2025)
- [Adaptive Benign Overfitting (ABO): Overparameterized RLS for Online Learning in Non-stationary Time-series](https://mlsanity.com/paper/2601.22200/) (2026)
- [Semi-Supervised Learning with Noisy Proxy Covariates: Generalization Bounds and Distribution Regression](https://mlsanity.com/paper/2606.00512/) (2026)

## Abstract (arXiv)

Modern machine learning often operates in the regime where the number of parameters is much higher than the number of data points, with zero training loss and yet good generalization, thereby contradicting the classical bias-variance trade-off. This textitbenign overfitting phenomenon has recently been characterized using so called textitdouble descent curves where the risk undergoes another descent (in addition to the classical U-shaped learning curve when the number of parameters is small) as we increase the number of parameters beyond a certain threshold. In this paper, we examine the conditions under which textitBenign Overfitting occurs in the random feature (RF) models, i.e. in a two-layer neural network with fixed first layer weights. We adopt a new view of random feature and show that textitbenign overfitting arises due to the noise which resides in such features (the noise may already be present in the data and propagate to the features or it may be added by the user to the features directly) and plays an important implicit regularization role in the phenomenon.