mlsanityData to 7 Oct 2026

A replica analysis of under-bagging

arXiv 2404.09779 ↗First posted 15 Apr 2024stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 4

Not on the map. Why?

Abstract (arXiv)

Under-bagging (UB), which combines under-sampling and bagging, is a popular ensemble learning method for training classifiers on an imbalanced data. Using bagging to reduce the increased variance caused by the reduction in sample size due to under-sampling is a natural approach. However, it has recently been pointed out that in generalized linear models, naive bagging, which does not consider the class imbalance structure, and ridge regularization can produce the same results. Therefore, it is not obvious whether it is better to use UB, which requires an increased computational cost proportional to the number of under-sampled data sets, when training linear models. Given such a situation, in this study, we heuristically derive a sharp asymptotics of UB and use it to compare with several other popular methods for learning from imbalanced data, in the scenario where a linear classifier is trained from a two-component mixture data. The methods compared include the under-sampling (US) method, which trains a model using a single realization of the under-sampled data, and the simple weighting (SW) method, which trains a model with a weighted loss on the entire data. It is shown that the performance of UB is improved by increasing the size of the majority class while keeping the size of the minority fixed, even though the class imbalance can be large, especially when the size of the minority class is small. This is in contrast to US, whose performance is almost independent of the majority class size. In this sense, bagging and simple regularization differ as methods to reduce the variance increased by under-sampling. On the other hand, the performance of SW with the optimal weighting coefficients is almost equal to UB, indicating that the combination of reweighting and regularization may be similar to UB.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 28 of its 61 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

28 of 61 references.

References · 28

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  3. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
  4. 2015Cost Sensitive Learning of Deep Feature Representations from Imbalanced Datacited by 167
  5. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  7. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  8. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  9. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 79
  10. 2013Confidence Intervals and Hypothesis Testing for High-Dimensional Regressioncited by 78
  11. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  12. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
16 more
  1. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  2. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  3. 2021A unifying tutorial on Approximate Message Passingcited by 49
  4. 2019The Implicit Regularization of Ordinary Least Squares Ensemblescited by 25
  5. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  6. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  7. 2013Hypothesis Testing in High-Dimensional Regression under the Gaussian Random Design Model: Asymptotic Theorycited by 18
  8. 2013Class Imbalance Problem in Data Mining Reviewcited by 16
  9. 2023Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validationcited by 11
  10. 2023Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuningcited by 11
  11. 2022Bias-inducing geometries: an exactly solvable data model with fairness implicationscited by 8
  12. 2022The Role of Pseudo-labels in Self-training Linear Classifiers on High-dimensional Gaussian Mixture Datacited by 7
  13. 2024Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regressioncited by 4
  14. 2018Semi-Analytic Resampling in Lassocited by 3
  15. 2019Replicated Vector Approximate Message Passing For Resampling Problemcited by 2
  16. 2020Semi-analytic approximate stability selection for correlated data in generalized linear modelscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025High-Dimensional Analysis of Bootstrap Ensemble Classifierscited by 3
  2. 2024Building Conformal Prediction Intervals with Approximate Message Passingcited by 2
  3. 2024When resampling/reweighting improves feature learning in imbalanced classification?: A toy-model studycited by 1
  4. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1