mlsanityData to 14 Oct 2026

Imbalance Trouble: Revisiting Neural-Collapse Geometry

arXiv 2208.05512 ↗First posted 10 Aug 2022cs.LG stat.MLcited by 80

Not on the map. Why?

Abstract (arXiv)

Neural Collapse refers to the remarkable structural properties characterizing the geometry of class embeddings and classifier weights, found by deep nets when trained beyond zero training error. However, this characterization only holds for balanced data. Here we thus ask whether it can be made invariant to class imbalances. Towards this end, we adopt the unconstrained-features model (UFM), a recent theoretical model for studying neural collapse, and introduce Simplex-Encoded-Labels Interpolation (SELI) as an invariant characterization of the neural collapse phenomenon. Specifically, we prove for the UFM with cross-entropy loss and vanishing regularization that, irrespective of class imbalances, the embeddings and classifiers always interpolate a simplex-encoded label matrix and that their individual geometries are determined by the SVD factors of this same label matrix. We then present extensive experiments on synthetic and real datasets that confirm convergence to the SELI geometry. However, we caution that convergence worsens with increasing imbalances. We theoretically support this finding by showing that unlike the balanced case, when minorities are present, ridge-regularization plays a critical role in tweaking the geometry. This defines new questions and motivates further investigations into the impact of class imbalances on the rates at which first-order methods converge to their asymptotically preferred solutions.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

33 of 51 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  4. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2020Long-tail learning via logit adjustmentcited by 618
  8. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  10. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
  11. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  12. 2020Directional convergence and alignment in deep learningcited by 178
21 more
  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  3. 2020Neural collapse with unconstrained featurescited by 142
  4. 2018Convergence of Gradient Descent on Separable Datacited by 110
  5. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  6. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  7. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  8. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  9. 2018Risk and parameter convergence of logistic regressioncited by 85
  10. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  11. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  12. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  13. 2020Gradient descent follows the regularization path for general lossescited by 59
  14. 2019Adjusting Decision Boundary for Class Imbalanced Learningcited by 52
  15. 2020Neural Collapse with Cross-Entropy Losscited by 52
  16. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  17. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  18. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  19. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  20. 2017Structured Low-Rank Matrix Factorization: Global Optimality, Algorithms, and Applicationscited by 16
  21. 2015Connections Between Nuclear Norm and Frobenius Norm Based Representationscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 22; 2024: 26; 2025: 16; 2026: 12.

80 papers.

Sort:

By area:

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  2. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  3. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  4. 2023Transformers as Support Vector Machinescited by 74
  5. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  6. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  7. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  8. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  9. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  10. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  11. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  12. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
38 more
  1. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  2. 2022Perturbation Analysis of Neural Collapsecited by 35
  3. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  4. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  5. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  6. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  7. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  8. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  9. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  10. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  11. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  12. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  13. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  14. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  15. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  16. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  17. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  18. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  19. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  20. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  21. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  22. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  23. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  24. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  25. 2023Engineering the Neural Collapse Geometry of Supervised-Contrastive Losscited by 8
  26. 2023Exploring Weight Balancing on Long-Tailed Recognition Problemcited by 7
  27. 2024Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Lossescited by 7
  28. 2022Pre-Training Representations of Binary Code Using Contrastive Learningcited by 6
  29. 2023Learning Prototype Classifiers for Long-Tailed Recognitioncited by 6
  30. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  31. 2023Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batchingcited by 5
  32. 2024Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapsecited by 5
  33. 2024Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Featurescited by 5
  34. 2024Enhancing Compositional Generalization via Compositional Feature Alignmentcited by 4
  35. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  36. 2024The Fair Language Model Paradoxcited by 4
  37. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  38. 2024Accurately Predicting Probabilities of Safety-Critical Rare Events for Intelligent Systemscited by 3

50 most cited of 80 All 80 (JSON)