mlsanityData to 14 Oct 2026

A Theoretical Analysis of the Learning Dynamics under Class Imbalance

arXiv 2207.00391 ↗First posted 1 Jul 2022stat.ML cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learning, showing that the learning curves for minority and majority classes follow sub-optimal trajectories when training with a gradient-based optimizer. This slowdown is related to the imbalance ratio and can be traced back to a competition between the optimization of different classes. Our main contribution is the analysis of the convergence of full-batch (GD) and stochastic gradient descent (SGD), and of variants that renormalize the contribution of each per-class gradient. We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient. With SGD, class imbalance has an additional effect on the direction of the gradients: the minority class suffers from a higher directional noise, which reduces the effectiveness of the per-class gradient normalization. Our findings not only allow us to understand the potential and limitations of strategies involving the per-class gradients, but also the reason for the effectiveness of previously used solutions for class imbalance such as oversampling.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

30 of 56 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  5. 2018Group Normalizationcited by 1,524
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  9. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  11. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  12. 2020Long-tail learning via logit adjustmentcited by 618
18 more
  1. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  2. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  3. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  4. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  5. 2017How to Escape Saddle Points Efficientlycited by 285
  6. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  7. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  8. 2019Does Object Recognition Work for Everyone?cited by 151
  9. 2020Remix: Rebalanced Mixupcited by 144
  10. 2022Long-Tailed Recognition via Weight Balancingcited by 107
  11. 2017The Devil is in the Tails: Fine-grained Classification in the Wildcited by 89
  12. 2018Escaping Saddles with Stochastic Gradientscited by 66
  13. 2018Imbalanced Deep Learning by Minority Class Incremental Rectificationcited by 65
  14. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 63
  15. 2020Why resampling outperforms reweighting for correcting sampling bias with stochastic gradientscited by 34
  16. 2021Procrustean Training for Imbalanced Deep Learningcited by 24
  17. 2021A Tale Of Two Long Tailscited by 19
  18. 2021Deep Learning Classification of Lake Zooplanktoncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 4; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  2. 2024Gradient Reweighting: Towards Imbalanced Class-Incremental Learningcited by 37
  3. 2024Restoring balance: principled under/oversampling of data for optimal classificationcited by 6
  4. 2023Initial Guessing Bias: How Untrained Networks Favor Some Classescited by 5
  5. 2024Early learning of the optimal constant solution in neural networks and humanscited by 3
  6. 2024Neuron Abandoning Attention Flow: Visual Explanation of Dynamics inside CNN Modelscited by 3
  7. 2024GradStop: Exploring Training Dynamics in Unsupervised Outlier Detection through Gradientcited by 3
  8. 2025Imbalanced Classification through the Lens of Spurious Correlationscited by 3
  9. 2023Understanding the Role of Layer Normalization in Label-Skewed Federated Learningcited by 2
  10. 2025GRASP: Guided Residual Adapters with Sample-wise Partitioningcited by 2
  11. 2023On The Relationship Between Continual Learning and Long-Tailed Recognitioncited by 1
  12. 2024When resampling/reweighting improves feature learning in imbalanced classification?: A toy-model studycited by 1
8 more
  1. 2025Class Imbalance in Anomaly Detection: Learning from an Exactly Solvable Modelcited by 1
  2. 2025When majority rules, minority loses: bias amplification of gradient descentcited by 1
  3. 2024Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approachcited by 0
  4. 2026A Theoretical and Empirical Taxonomy of Imbalance in Binary Classificationcited by 0
  5. 2026Visibility nowcasting in South Korea: a machine learning approach to class imbalance and distribution shiftcited by 0
  6. 2026SEED: Semi-supervised Continual MalwarE Detection for Tackling ConcEpt Drift on a BuDgetcited by 0
  7. 2026Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learningcited by 0
  8. 2026Bounded Adjustment with Reliability-Guided Embedding for Imbalanced Learning with Noisy Labelscited by 0