# Bagging in overparameterized learning: Risk characterization and risk monotonization

arXiv: https://arxiv.org/abs/2210.11445 · HTML: https://mlsanity.com/paper/2210.11445/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2022.

## Built on by (most cited first, 11 of 11)

- [Bagging Provides Assumption-free Stability](https://mlsanity.com/paper/2301.12600/) (2023)
- [Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation](https://mlsanity.com/paper/2304.13016/) (2023)
- [Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning](https://mlsanity.com/paper/2310.04357/) (2023)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Corrected generalized cross-validation for finite ensembles of penalized estimators](https://mlsanity.com/paper/2310.01374/) (2023)
- [Extrapolated cross-validation for randomized ensembles](https://mlsanity.com/paper/2302.13511/) (2023)
- [Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression](https://mlsanity.com/paper/2402.13622/) (2024)
- [Implicit Regularization Paths of Weighted Neural Representations](https://mlsanity.com/paper/2408.15784/) (2024)
- [No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions](https://mlsanity.com/paper/2412.05418/) (2024)
- [Understanding the Role of Optimization in Double Descent](https://mlsanity.com/paper/2312.03951/) (2023)
- [Assumption-Lean Post-Integrated Inference with Surrogate Control Outcomes](https://mlsanity.com/paper/2410.04996/) (2024)

## Abstract (arXiv)

Bagging is a commonly used ensemble technique in statistics and machine learning to improve the performance of prediction procedures. In this paper, we study the prediction risk of variants of bagged predictors under the proportional asymptotics regime, in which the ratio of the number of features to the number of observations converges to a constant. Specifically, we propose a general strategy to analyze the prediction risk under squared error loss of bagged predictors using classical results on simple random sampling. Specializing the strategy, we derive the exact asymptotic risk of the bagged ridge and ridgeless predictors with an arbitrary number of bags under a well-specified linear model with arbitrary feature covariance matrices and signal vectors. Furthermore, we prescribe a generic cross-validation procedure to select the optimal subsample size for bagging and discuss its utility to eliminate the non-monotonic behavior of the limiting risk in the sample size (i.e., double or multiple descents). In demonstrating the proposed procedure for bagged ridge and ridgeless predictors, we thoroughly investigate the oracle properties of the optimal subsample size and provide an in-depth comparison between different bagging variants.