# Precise Asymptotics of Bagging Regularized M-estimators

arXiv: https://arxiv.org/abs/2409.15252 · HTML: https://mlsanity.com/paper/2409.15252/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2024.

## Built on by (most cited first, 4 of 4)

- [High-Dimensional Analysis of Bootstrap Ensemble Classifiers](https://mlsanity.com/paper/2505.14587/) (2025)
- [Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learning](https://mlsanity.com/paper/2410.01259/) (2024)
- [Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference](https://mlsanity.com/paper/2501.02624/) (2025)
- [Transfer Learning in ℓ1 Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis](https://mlsanity.com/paper/2409.17704/) (2024)

## Abstract (arXiv)

We characterize the squared prediction risk of ensemble estimators obtained through subagging (subsample bootstrap aggregating) regularized M-estimators and construct a consistent estimator for the risk. Specifically, we consider a heterogeneous collection of M ≥ 1 regularized M-estimators, each trained with (possibly different) subsample sizes, convex differentiable losses, and convex regularizers. We operate under the proportional asymptotics regime, where the sample size n, feature size p, and subsample sizes km for m ∈ [M] all diverge with fixed limiting ratios n/p and km/n. Key to our analysis is a new result on the joint asymptotic behavior of correlations between the estimator and residual errors on overlapping subsamples, governed through a (provably) contractive nonlinear system of equations. Of independent interest, we also establish convergence of trace functionals related to degrees of freedom in the non-ensemble setting (with M = 1) along the way, extending previously known cases for squared loss with ridge and lasso regularizers. When specialized to homogeneous ensembles trained with a common loss, regularizer, and subsample size, the risk characterization sheds some light on the implicit regularization effect due to the ensemble and subsample sizes (M,k). For any ensemble size M, optimally tuning subsample size yields sample-wise monotonic risk. For the full-ensemble estimator (when M → ∞), the optimal subsample size k^star tends to be in the overparameterized regime (k^star ≤ minn,p), when explicit regularization is vanishing. Finally, joint optimization of subsample size, ensemble size, and regularization can significantly outperform regularizer optimization alone on the full data (without any subagging).