mlsanityData to 14 Oct 2026

How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance

arXiv 2403.07310 ↗First posted 12 Mar 2024stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Group imbalance has been a known problem in empirical risk minimization (ERM), where the achieved high average accuracy is accompanied by low accuracy in a minority group. Despite algorithmic efforts to improve the minority group accuracy, a theoretical generalization analysis of ERM on individual groups remains elusive. By formulating the group imbalance problem with the Gaussian Mixture Model, this paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance. Although our theoretical framework is centered on binary classification using a one-hidden-layer neural network, to the best of our knowledge, we provide the first theoretical analysis of the group-level generalization of ERM in addition to the commonly studied average generalization performance. Sample insights of our theoretical results include that when all group-level co-variance is in the medium regime and all mean are close to zero, the learning performance is most desirable in the sense of a small sample complexity, a fast training rate, and a high average and group-level testing accuracy. Moreover, we show that increasing the fraction of the minority group in the training data does not necessarily improve the generalization performance of the minority group. Our theoretical results are validated on both synthetic and empirical datasets, such as CelebA and CIFAR-10 in image classification.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 59 of its 96 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

59 of 96 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
38 more
  1. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  2. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  3. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
  4. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  5. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 151
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  10. 2020Feature Space Augmentation for Long-Tailed Datacited by 136
  11. 2020M2m: Imbalanced Classification via Major-to-minor Translationcited by 134
  12. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  13. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  14. 2021The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classificationcited by 100
  15. 2021MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognitioncited by 99
  16. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  17. 2017Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutionscited by 93
  18. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  19. 2017Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classificationcited by 85
  20. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
  21. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  22. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  23. 2019Generative Adversarial Minority Oversamplingcited by 73
  24. 2021Subgroup Generalization and Fairness of Graph Neural Networkscited by 69
  25. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  26. 2018Deep Imbalanced Attribute Classification using Visual Attention Aggregationcited by 56
  27. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  28. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  29. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  30. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  31. 2018Singularity, Misspecification, and the Convergence Rate of EMcited by 38
  32. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  33. 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 32
  34. 2017On Learning Mixtures of Well-Separated Gaussianscited by 30
  35. 2020Fast Learning of Graph Neural Networks with Guaranteed Generalizability: One-hidden-layer Casecited by 24
  36. 2019Sharp Analysis of Expectation-Maximization for Weakly Identifiable Modelscited by 23
  37. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 23
  38. 2022Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Samplingcited by 23

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  2. 2024What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encodingcited by 14
  3. 2024Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysiscited by 8
  4. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  5. 2025FairNet: Dynamic Fairness Correction without Performance Loss via Contrastive Conditional LoRAcited by 1
  6. 2025Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsificationcited by 0