# When Does ℓ2-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ℓ1 Implicit Bias

arXiv: https://arxiv.org/abs/2605.06314 · HTML: https://mlsanity.com/paper/2605.06314/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 71 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 30 of 30)

- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [Just Interpolate: Kernel "Ridgeless" Regression Can Generalize](https://mlsanity.com/paper/1808.00387/) (2018)
- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Benign Overfitting in Two-layer Convolutional Neural Networks](https://mlsanity.com/paper/2202.06526/) (2022)
- [Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime](https://mlsanity.com/paper/2004.12019/) (2020)
- [Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting](https://mlsanity.com/paper/2106.09276/) (2021)
- [The Lasso with general Gaussian designs with applications to hypothesis testing](https://mlsanity.com/paper/2007.13716/) (2020)
- [Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting](https://mlsanity.com/paper/2207.06569/) (2022)
- [Margins, Shrinkage, and Boosting](https://mlsanity.com/paper/1303.4172/) (2013)
- [Benign Overfitting for Two-layer ReLU Convolutional Neural Networks](https://mlsanity.com/paper/2303.04145/) (2023)
- [Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization](https://mlsanity.com/paper/2409.19345/) (2024)
- [Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context](https://mlsanity.com/paper/2410.01774/) (2024)
- [Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions](https://mlsanity.com/paper/2006.08917/) (2020)
- [Benign Overfitting in Single-Head Attention](https://mlsanity.com/paper/2410.07746/) (2024)
- [Benign Overfitting in Deep Neural Networks under Lazy Training](https://mlsanity.com/paper/2305.19377/) (2023)
- [Benign Overfitting in Token Selection of Attention Mechanism](https://mlsanity.com/paper/2409.17625/) (2024)
- [Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression](https://mlsanity.com/paper/2510.15337/) (2025)
- [Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers](https://mlsanity.com/paper/2410.19139/) (2024)
- [Rethinking Benign Overfitting in Two-Layer Neural Networks](https://mlsanity.com/paper/2502.11893/) (2025)
- [ITBoost: Information-Theoretic Trust for Robust Boosting](https://mlsanity.com/paper/2605.04671/) (2026)

## Built on by (most cited first, 1 of 1)

- [Reducing Learner Redundancy in Boosting via Residual Orthogonalization](https://mlsanity.com/paper/2606.17567/) (2026)

## Abstract (arXiv)

Benign overfitting is well-characterized in ℓ2 geometries, but its behavior under the ℓ1 implicit bias of greedy ensembles remains challenging. The analytical barrier stems from the non-linear coupling of coordinate selection thresholds, which invalidates standard spectral resolvent tools. To isolate this algorithmic bias, we characterize the high-dimensional risk of continuous-time ℓ2-Boosting over p features and n samples. By coupling the Convex Gaussian Minimax Theorem with delicate asymptotic expansions of double-sided truncated Gaussian moments, we analytically resolve the non-smooth ℓ1 interpolant. Under an isotropic pure-noise model, we prove that benign overfitting fails at the linear rate: greedy selection localizes noise into sparse active sets, and the excess variance decays at a logarithmic rate Theta(σ2/log(p/n)) for noise variance σ2. We remark that while this localization mechanism should persist in the presence of signals, the exact signal-noise decomposition remains an open problem. For spiked-isotropic designs with k^* head eigenvalues and r2 = p - k^* tail dimensions, the risk converges to zero when r2 gg n, but only at a logarithmic rate Theta(σ2/log(r2/n)), which is slower than the linear decay observed in ℓ2 geometries. To avoid this slow convergence, we analyze the non-smooth subdifferential dynamics of the boosting flow. This yields a tuning-free early stopping rule that, under a bounded ℓ1-path condition, recovers the Lasso basic inequality and attains the minimax-optimal empirical prediction rate for ℓ1-bounded signals.