mlsanityData to 14 Oct 2026

Generalized Neural Collapse for a Large Number of Classes

arXiv 2310.05351 ↗First posted 9 Oct 2023cs.LG cs.AI cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Neural collapse provides an elegant mathematical characterization of learned last layer representations (a.k.a. features) and classifier weights in deep classification models. Such results not only provide insights but also motivate new techniques for improving practical deep models. However, most of the existing empirical and theoretical studies in neural collapse focus on the case that the number of classes is small relative to the dimension of the feature space. This paper extends neural collapse to cases where the number of classes are much larger than the dimension of feature space, which broadly occur for language models, retrieval systems, and face recognition applications. We show that the features and classifier exhibit a generalized neural collapse phenomenon, where the minimum one-vs-rest margins is maximized.We provide empirical study to verify the occurrence of generalized neural collapse in practical deep neural networks. Moreover, we provide theoretical study to show that the generalized neural collapse provably occurs under unconstrained feature model with spherical constraint, under certain technical conditions on feature dimension and number of classes.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 42 of its 75 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

42 of 75 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2021Convex optimizationcited by 2,515
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  10. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  11. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 971
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
30 more
  1. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  2. 2018Additive Margin Softmax for Face Verificationcited by 278
  3. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  7. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  8. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 142
  9. 2020Neural collapse with unconstrained featurescited by 142
  10. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  11. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  12. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  13. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  14. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  15. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  16. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  17. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  18. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  19. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  20. 2020Neural Collapse with Cross-Entropy Losscited by 52
  21. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  22. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  23. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  24. 2022Perturbation Analysis of Neural Collapsecited by 35
  25. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  26. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  27. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  28. 2022Continual Learning by Modeling Intra-Class Variationcited by 12
  29. 2023A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalizationcited by 7
  30. 2022Learning Towards the Largest Marginscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 11; 2026: 11.

39 papers.

Sort:

By area:

  1. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  2. 2024Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learningcited by 26
  3. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  4. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  5. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  6. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  7. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  8. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  9. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  10. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  11. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  12. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
27 more
  1. 2024Cell-ontology guided transcriptome foundation modelcited by 7
  2. 2025DualCP: Rehearsal-Free Domain-Incremental Learning via Dual-Level Concept Prototypecited by 7
  3. 2024Task structure and nonlinearity jointly determine learned representational geometrycited by 6
  4. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 6
  5. 2025Consistent Supervised-Unsupervised Alignment for Generalized Category Discoverycited by 6
  6. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  7. 2023Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamicscited by 3
  8. 2024All-around Neural Collapse for Imbalanced Classificationcited by 3
  9. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  10. 2025Mitigating Negative Flips via Margin Preserving Trainingcited by 3
  11. 2024Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)cited by 2
  12. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  13. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  14. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  15. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  16. 2025BCE vs. CE in Deep Feature Learningcited by 1
  17. 2025Heads collapse, features stay: Why Replay needs big bufferscited by 1
  18. 2026Optimizer choice matters for the emergence of Neural Collapsecited by 1
  19. 2026Neural collapse in the orthoplex regimecited by 1
  20. 2026A Stationary (and Therefore Compatible) Representation is All You Needcited by 1
  21. 2026XtraLight-MedMamba for Classification of Neoplastic Tubular Adenomascited by 0
  22. 2026Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recyclingcited by 0
  23. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  24. 2026The role of class encoding in neural collapsecited by 0
  25. 2026Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal imagescited by 0
  26. 2026Neural Collapse Is Forbidden: Information Floors in Language Modelscited by 0
  27. 2026Approximate Homomorphisms and Convergent Representations in Transducerscited by 0