mlsanityData to 7 Oct 2026

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?

arXiv 2203.09081 ↗First posted 17 Mar 2022cs.LG cs.CVcited by 100

Not on the map. Why?

Abstract (arXiv)

Modern deep neural networks for classification usually jointly learn a backbone for representation and a linear classifier to output the logit of each class. A recent study has shown a phenomenon called neural collapse that the within-class means of features and the classifier vectors converge to the vertices of a simplex equiangular tight frame (ETF) at the terminal phase of training on a balanced dataset. Since the ETF geometric structure maximally separates the pair-wise angles of all classes in the classifier, it is natural to raise the question, why do we spend an effort to learn a classifier when we know its optimal geometric structure? In this paper, we study the potential of learning a neural network for classification with the classifier randomly initialized as an ETF and fixed during training. Our analytical work based on the layer-peeled model indicates that the feature learning with a fixed ETF classifier naturally leads to the neural collapse state even when the dataset is imbalanced among classes. We further show that in this case the cross entropy (CE) loss is not necessary and can be replaced by a simple squared loss that shares the same global optimality but enjoys a better convergence property. Our experimental results show that our method is able to bring significant improvements with faster convergence on multiple imbalanced datasets.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

34 of 56 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  8. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2014Distributed Representations of Sentences and Documentscited by 677
22 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  3. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  4. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  7. 2020Neural collapse with unconstrained featurescited by 140
  8. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  9. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  10. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  11. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  12. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  13. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  14. 2020Neural Collapse with Cross-Entropy Losscited by 52
  15. 2018Fix your classifier: the marginal value of training the last weight layercited by 43
  16. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  17. 2020Separation and Concentration in Deep Networkscited by 28
  18. 2021Regular Polytope Networkscited by 24
  19. 2018Convolutional Neural Networks with Alternately Updated Cliquecited by 14
  20. 2018Optimization Algorithm Inspired Deep Neural Network Structure Designcited by 14
  21. 2017Building a Regular Decision Boundary with Deep Networkscited by 11
  22. 2019Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Familiescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 37; 2025: 23; 2026: 25.

100 papers.

Sort:

By area:

  1. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  2. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  3. 2023NECO: NEural Collapse Based Out-of-distribution detectioncited by 41
  4. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  5. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  6. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  7. 2022Problem-Dependent Power of Quantum Neural Networks on Multi-Class Classificationcited by 20
  8. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  9. 2023Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancercited by 17
  10. 2024An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learningcited by 17
  11. 2024CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignmentcited by 16
  12. 2025Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Datacited by 15
38 more
  1. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  2. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  3. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  4. 2023Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severitycited by 13
  5. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  6. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  7. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  8. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  9. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  10. 2024Classes Are Not Equal: An Empirical Study on Image Recognition Fairnesscited by 10
  11. 2025Improved Balanced Classification with Theoretically Grounded Loss Functionscited by 9
  12. 2023Engineering the Neural Collapse Geometry of Supervised-Contrastive Losscited by 8
  13. 2024Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacementscited by 8
  14. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
  15. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8
  16. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  17. 2023A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalizationcited by 7
  18. 2023Exploring Weight Balancing on Long-Tailed Recognition Problemcited by 7
  19. 2024Pushing Boundaries: Mixup's Influence on Neural Collapsecited by 7
  20. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7
  21. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  22. 2024Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretizationcited by 7
  23. 2023Towards Demystifying the Generalization Behaviors When Neural Collapse Emergescited by 6
  24. 2024Adaptive Parametric Activation: Unifying and Generalising Activation Functions Across Taskscited by 6
  25. 2023Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batchingcited by 5
  26. 2025A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasetscited by 5
  27. 2025LIFT+: Lightweight Fine-Tuning for Long-Tail Learningcited by 5
  28. 2025Consistent Supervised-Unsupervised Alignment for Generalized Category Discoverycited by 5
  29. 2023Pairwise Similarity Learning is SimPLEcited by 4
  30. 2024Enhancing Compositional Generalization via Compositional Feature Alignmentcited by 4
  31. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  32. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  33. 2024On Partial Prototype Collapse in the DINO Family of Self-Supervised Methodscited by 4
  34. 2024Dual-Head Knowledge Distillation: Enhancing Logits Utilization with an Auxiliary Headcited by 4
  35. 2025BatStyler: Advancing Multi-category Style Generation for Source-free Domain Generalizationcited by 4
  36. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  37. 2025Rethinking Continual Learning with Progressive Neural Collapsecited by 4
  38. 2024Hyperspherical Classification with Dynamic Label-to-Prototype Assignmentcited by 3

50 most cited of 100 All 100 (JSON)