mlsanityData to 7 Oct 2026

Prevalence of Neural Collapse during the terminal phase of deep learning training

arXiv 2008.08186 ↗First posted 18 Aug 2020cs.LG cs.CV stat.MLcited by 713

Not on the map. Why?

Abstract (arXiv)

Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is pushed towards zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call Neural Collapse, involving four deeply interconnected phenomena: (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class-means; (NC2) The class-means collapse to the vertices of a Simplex Equiangular Tight Frame (ETF); (NC3) Up to rescaling, the last-layer classifiers collapse to the class-means, or in other words to the Simplex ETF, i.e. to a self-dual configuration; (NC4) For a given activation, the classifier's decision collapses to simply choosing whichever class has the closest train class-mean, i.e. the Nearest Class Center (NCC) decision rule. The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability.

Key facts. Cited by 713 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 63; 2023: 104; 2024: 143; 2025: 167; 2026: 203.

713 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 362
  2. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  3. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  5. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  6. 2025Hierarchical Reasoning Modelcited by 167
  7. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  8. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  9. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  10. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 143
  11. 2023The emergence of clusters in self-attention dynamicscited by 133
  12. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
38 more
  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  3. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  4. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  5. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  6. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  7. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  8. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  9. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  10. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  11. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  12. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  13. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  14. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  15. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  16. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  17. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  18. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  19. 2021A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challengescited by 68
  20. 2023Transformers as Support Vector Machinescited by 67
  21. 2021Dissecting Supervised Contrastive Learningcited by 64
  22. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  23. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  24. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  25. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  26. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  27. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  28. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  29. 2024Deep Networks Always Grok and Here is Whycited by 49
  30. 2023Fast Decision Boundary based Out-of-Distribution Detectorcited by 48
  31. 2025BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learningcited by 48
  32. 2023Rethinking Domain Generalization for Face Anti-spoofing: Separability and Alignmentcited by 47
  33. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  34. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  35. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  36. 2021Formalizing and Estimating Distribution Inference Riskscited by 42
  37. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  38. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41

50 most cited of 713 All 713 (JSON)