# Generalized Neural Collapse for a Large Number of Classes

arXiv: https://arxiv.org/abs/2310.05351 · HTML: https://mlsanity.com/paper/2310.05351/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 75 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2302.03004/) (2023)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra](https://mlsanity.com/paper/2008.11865/) (2020)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)

## Built on by (most cited first, 30 of 39)

- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning](https://mlsanity.com/paper/2412.07909/) (2024)
- [Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes](https://mlsanity.com/paper/2410.23126/) (2024)
- [Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?](https://mlsanity.com/paper/2405.14468/) (2024)
- [Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse](https://mlsanity.com/paper/2410.04887/) (2024)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)
- [A Global Geometric Analysis of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2406.01909/) (2024)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Cross Entropy versus Label Smoothing: A Neural Collapse Perspective](https://mlsanity.com/paper/2402.03979/) (2024)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers](https://mlsanity.com/paper/2505.15239/) (2025)
- [Control-oriented Clustering of Visual Latent Representation](https://mlsanity.com/paper/2410.05063/) (2024)
- [Cell-ontology guided transcriptome foundation model](https://mlsanity.com/paper/2408.12373/) (2024)
- [DualCP: Rehearsal-Free Domain-Incremental Learning via Dual-Level Concept Prototype](https://mlsanity.com/paper/2503.18042/) (2025)
- [Task structure and nonlinearity jointly determine learned representational geometry](https://mlsanity.com/paper/2401.13558/) (2024)
- [From Compression to Expression: A Layerwise Analysis of In-Context Learning](https://mlsanity.com/paper/2505.17322/) (2025)
- [Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery](https://mlsanity.com/paper/2507.04725/) (2025)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics](https://mlsanity.com/paper/2311.05061/) (2023)
- [All-around Neural Collapse for Imbalanced Classification](https://mlsanity.com/paper/2408.07253/) (2024)
- [Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective](https://mlsanity.com/paper/2410.23391/) (2024)
- [Mitigating Negative Flips via Margin Preserving Training](https://mlsanity.com/paper/2511.08322/) (2025)
- [Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)](https://mlsanity.com/paper/2403.18699/) (2024)
- [Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model](https://mlsanity.com/paper/2505.09308/) (2025)
- [Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data](https://mlsanity.com/paper/2510.21078/) (2025)
- [Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics](https://mlsanity.com/paper/2512.04006/) (2025)
- [Structure Before Collapse: Transient semantic geometry in next-token prediction](https://mlsanity.com/paper/2606.26749/) (2026)
- [BCE vs. CE in Deep Feature Learning](https://mlsanity.com/paper/2505.05813/) (2025)
- [Heads collapse, features stay: Why Replay needs big buffers](https://mlsanity.com/paper/2512.07400/) (2025)
- [Optimizer choice matters for the emergence of Neural Collapse](https://mlsanity.com/paper/2602.16642/) (2026)

## Abstract (arXiv)

Neural collapse provides an elegant mathematical characterization of learned last layer representations (a.k.a. features) and classifier weights in deep classification models. Such results not only provide insights but also motivate new techniques for improving practical deep models. However, most of the existing empirical and theoretical studies in neural collapse focus on the case that the number of classes is small relative to the dimension of the feature space. This paper extends neural collapse to cases where the number of classes are much larger than the dimension of feature space, which broadly occur for language models, retrieval systems, and face recognition applications. We show that the features and classifier exhibit a generalized neural collapse phenomenon, where the minimum one-vs-rest margins is maximized.We provide empirical study to verify the occurrence of generalized neural collapse in practical deep neural networks. Moreover, we provide theoretical study to show that the generalized neural collapse provably occurs under unconstrained feature model with spherical constraint, under certain technical conditions on feature dimension and number of classes.