# Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression arXiv: https://arxiv.org/abs/2402.13622 · HTML: https://mlsanity.com/paper/2402.13622/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 77 references that are in the map. - First posted on arXiv 21 Feb 2024. ## Builds on (most cited first, 29 of 29) - [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015) - [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021) - [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017) - [Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime](https://mlsanity.com/paper/2003.01054/) (2020) - [A modern maximum-likelihood theory for high-dimensional logistic regression](https://mlsanity.com/paper/1803.06964/) (2018) - [Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition](https://mlsanity.com/paper/2011.03321/) (2020) - [The committee machine: Computational to statistical gaps in learning a two-layers neural network](https://mlsanity.com/paper/1806.05451/) (2018) - [Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions](https://mlsanity.com/paper/2106.03791/) (2021) - [The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression](https://mlsanity.com/paper/1804.09753/) (2018) - [A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifiers](https://mlsanity.com/paper/2002.01586/) (2020) - [Phase retrieval in high dimensions: Statistical and computational phase transitions](https://mlsanity.com/paper/2006.05228/) (2020) - [Graph-based Approximate Message Passing Iterations](https://mlsanity.com/paper/2109.11905/) (2021) - [What causes the test error? Going beyond bias-variance via ANOVA](https://mlsanity.com/paper/2010.05170/) (2020) - [The Implicit Regularization of Ordinary Least Squares Ensembles](https://mlsanity.com/paper/1910.04743/) (2019) - [Generalization Error of Generalized Linear Models in High Dimensions](https://mlsanity.com/paper/2005.00180/) (2020) - [Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)](https://mlsanity.com/paper/2006.06581/) (2020) - [Theoretical characterization of uncertainty in high-dimensional linear classification](https://mlsanity.com/paper/2202.03295/) (2022) - [On double-descent in uncertainty quantification in overparametrized models](https://mlsanity.com/paper/2210.12760/) (2022) - [The Likelihood Ratio Test in High-Dimensional Logistic Regression Is Asymptotically a Rescaled Chi-Square](https://mlsanity.com/paper/1706.01191/) (2017) - [Out-of-sample error estimate for robust M-estimators with convex penalty](https://mlsanity.com/paper/2008.11840/) (2020) - [Bagging in overparameterized learning: Risk characterization and risk monotonization](https://mlsanity.com/paper/2210.11445/) (2022) - [Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation](https://mlsanity.com/paper/2304.13016/) (2023) - [Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning](https://mlsanity.com/paper/2310.04357/) (2023) - [Understanding the Under-Coverage Bias in Uncertainty Estimation](https://mlsanity.com/paper/2106.05515/) (2021) - [Expectation consistency for calibration of neural networks](https://mlsanity.com/paper/2303.02644/) (2023) - [Sketched Ridgeless Linear Regression: The Role of Downsampling](https://mlsanity.com/paper/2302.01088/) (2023) - [Asymptotic normality of robust M-estimators with convex penalty](https://mlsanity.com/paper/2107.03826/) (2021) - [Semi-Analytic Resampling in Lasso](https://mlsanity.com/paper/1802.10254/) (2018) - [Replicated Vector Approximate Message Passing For Resampling Problem](https://mlsanity.com/paper/1905.09545/) (2019) ## Built on by (most cited first, 4 of 4) - [A replica analysis of under-bagging](https://mlsanity.com/paper/2404.09779/) (2024) - [Building Conformal Prediction Intervals with Approximate Message Passing](https://mlsanity.com/paper/2410.16493/) (2024) - [Implicit Regularization Paths of Weighted Neural Representations](https://mlsanity.com/paper/2408.15784/) (2024) - [Modeling Sampling Distributions of Test Statistics with Autograd](https://mlsanity.com/paper/2405.02488/) (2024) ## Abstract (arXiv) We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dimensional supervised regression tasks. We provide a tight asymptotic description of the biases and variances estimated by these methods in the context of generalized linear models, such as ridge and logistic regression, taking the limit where the number of samples n and dimension d of the covariates grow at a comparable fixed rate α!=! n/d. Our findings are three-fold: i) resampling methods are fraught with problems in high dimensions and exhibit the double-descent-like behavior typical of these situations; ii) only when α is large enough do they provide consistent and reliable error estimations (we give convergence rates); iii) in the over-parametrized regime α!