# A Theoretical Analysis of the Learning Dynamics under Class Imbalance arXiv: https://arxiv.org/abs/2207.00391 ยท HTML: https://mlsanity.com/paper/2207.00391/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 56 references that are in the map. - First posted on arXiv 1 Jul 2022. ## Builds on (most cited first, 30 of 30) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019) - [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018) - [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020) - [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018) - [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019) - [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017) - [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013) - [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018) - [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020) - [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016) - [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019) - [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019) - [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020) - [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017) - [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020) - [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020) - [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019) - [Remix: Rebalanced Mixup](https://mlsanity.com/paper/2007.03943/) (2020) - [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022) - [The Devil is in the Tails: Fine-grained Classification in the Wild](https://mlsanity.com/paper/1709.01450/) (2017) - [Escaping Saddles with Stochastic Gradients](https://mlsanity.com/paper/1803.05999/) (2018) - [Imbalanced Deep Learning by Minority Class Incremental Rectification](https://mlsanity.com/paper/1804.10851/) (2018) - [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022) - [Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients](https://mlsanity.com/paper/2009.13447/) (2020) - [Procrustean Training for Imbalanced Deep Learning](https://mlsanity.com/paper/2104.01769/) (2021) - [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021) - [Deep Learning Classification of Lake Zooplankton](https://mlsanity.com/paper/2108.05258/) (2021) ## Built on by (most cited first, 20 of 20) - [Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models](https://mlsanity.com/paper/2402.19449/) (2024) - [Gradient Reweighting: Towards Imbalanced Class-Incremental Learning](https://mlsanity.com/paper/2402.18528/) (2024) - [Restoring balance: principled under/oversampling of data for optimal classification](https://mlsanity.com/paper/2405.09535/) (2024) - [Initial Guessing Bias: How Untrained Networks Favor Some Classes](https://mlsanity.com/paper/2306.00809/) (2023) - [Early learning of the optimal constant solution in neural networks and humans](https://mlsanity.com/paper/2406.17467/) (2024) - [Neuron Abandoning Attention Flow: Visual Explanation of Dynamics inside CNN Models](https://mlsanity.com/paper/2412.01202/) (2024) - [GradStop: Exploring Training Dynamics in Unsupervised Outlier Detection through Gradient](https://mlsanity.com/paper/2412.08501/) (2024) - [Imbalanced Classification through the Lens of Spurious Correlations](https://mlsanity.com/paper/2510.27650/) (2025) - [Understanding the Role of Layer Normalization in Label-Skewed Federated Learning](https://mlsanity.com/paper/2308.09565/) (2023) - [GRASP: Guided Residual Adapters with Sample-wise Partitioning](https://mlsanity.com/paper/2512.01675/) (2025) - [On The Relationship Between Continual Learning and Long-Tailed Recognition](https://mlsanity.com/paper/2306.13275/) (2023) - [When resampling/reweighting improves feature learning in imbalanced classification?: A toy-model study](https://mlsanity.com/paper/2409.05598/) (2024) - [Class Imbalance in Anomaly Detection: Learning from an Exactly Solvable Model](https://mlsanity.com/paper/2501.11638/) (2025) - [When majority rules, minority loses: bias amplification of gradient descent](https://mlsanity.com/paper/2505.13122/) (2025) - [Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach](https://mlsanity.com/paper/2406.07332/) (2024) - [A Theoretical and Empirical Taxonomy of Imbalance in Binary Classification](https://mlsanity.com/paper/2601.04149/) (2026) - [Visibility nowcasting in South Korea: a machine learning approach to class imbalance and distribution shift](https://mlsanity.com/paper/2605.21507/) (2026) - [SEED: Semi-supervised Continual MalwarE Detection for Tackling ConcEpt Drift on a BuDget](https://mlsanity.com/paper/2605.24903/) (2026) - [Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning](https://mlsanity.com/paper/2607.01474/) (2026) - [Bounded Adjustment with Reliability-Guided Embedding for Imbalanced Learning with Noisy Labels](https://mlsanity.com/paper/2609.16380/) (2026) ## Abstract (arXiv) Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learning, showing that the learning curves for minority and majority classes follow sub-optimal trajectories when training with a gradient-based optimizer. This slowdown is related to the imbalance ratio and can be traced back to a competition between the optimization of different classes. Our main contribution is the analysis of the convergence of full-batch (GD) and stochastic gradient descent (SGD), and of variants that renormalize the contribution of each per-class gradient. We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient. With SGD, class imbalance has an additional effect on the direction of the gradients: the minority class suffers from a higher directional noise, which reduces the effectiveness of the per-class gradient normalization. Our findings not only allow us to understand the potential and limitations of strategies involving the per-class gradients, but also the reason for the effectiveness of previously used solutions for class imbalance such as oversampling.