mlsanityData to 7 Oct 2026

Neural Collapse: A Review on Modelling Principles and Generalization

arXiv 2206.04041 ↗First posted 8 Jun 2022cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

Deep classifier neural networks enter the terminal phase of training (TPT) when training error reaches zero and tend to exhibit intriguing Neural Collapse (NC) properties. Neural collapse essentially represents a state at which the within-class variability of final hidden layer outputs is infinitesimally small and their class means form a simplex equiangular tight frame. This simplifies the last layer behaviour to that of a nearest-class center decision rule. Despite the simplicity of this state, the dynamics and implications of reaching it are yet to be fully understood. In this work, we review the principles which aid in modelling neural collapse, followed by the implications of this state on generalization and transfer learning capabilities of neural networks. Finally, we conclude by discussing potential avenues and directions for future research.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 108 of its 178 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

108 of 178 references.

References · 108

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2016Wide Residual Networkscited by 3,057
38 more
  1. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  2. 2020Supervised Contrastive Learningcited by 2,803
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  5. 2018Group Normalizationcited by 1,514
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  10. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  11. 2021Scaling Vision Transformerscited by 993
  12. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  13. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  14. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  15. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  16. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  17. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
  18. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  19. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  20. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  21. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  22. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  23. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  24. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  25. 2019Benign Overfitting in Linear Regressioncited by 544
  26. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  27. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  28. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  29. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  30. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  31. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 402
  32. 2020Understanding the Behaviour of Contrastive Losscited by 397
  33. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  34. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  35. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  36. 2016Large-Margin Softmax Loss for Convolutional Neural Networkscited by 296
  37. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  38. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 19; 2024: 31; 2025: 24; 2026: 14.

95 papers.

Sort:

By area:

  1. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  2. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  3. 2025BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learningcited by 48
  4. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  5. 2023NECO: NEural Collapse Based Out-of-distribution detectioncited by 41
  6. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  7. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  8. 2022Perturbation Analysis of Neural Collapsecited by 34
  9. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 32
  10. 2025Superposition Yields Robust Neural Scalingcited by 30
  11. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  12. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
38 more
  1. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  2. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  3. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  4. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  5. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  6. 2023Neural (Tangent Kernel) Collapsecited by 18
  7. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  8. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  9. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  10. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  11. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  12. 2024Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detectioncited by 12
  13. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  14. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  15. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  16. 2024Formation of Representations in Neural Networkscited by 10
  17. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  18. 2024What Is Missing In Homophily? Disentangling Graph Homophily For Graph Neural Networkscited by 9
  19. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 9
  20. 2024BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Rangescited by 8
  21. 2024Pushing Boundaries: Mixup's Influence on Neural Collapsecited by 7
  22. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  23. 2025Adversarial Attacks Leverage Interference Between Features in Superpositioncited by 7
  24. 2023Towards Demystifying the Generalization Behaviors When Neural Collapse Emergescited by 6
  25. 2024Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Lossescited by 6
  26. 2024Task structure and nonlinearity jointly determine learned representational geometrycited by 5
  27. 2024Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapsecited by 5
  28. 2025Why and How Auxiliary Tasks Improve JEPA Representationscited by 5
  29. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  30. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  31. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  32. 2025Connecting Neural Models Latent Geometries with Relative Geodesic Representationscited by 4
  33. 2023DRAUC: An Instance-wise Distributionally Robust AUC Optimization Frameworkcited by 3
  34. 2023On the Dynamics Under the Unhinged Loss and Beyondcited by 3
  35. 2024The Proof of Kolmogorov-Arnold May Illuminate Neural Network Learningcited by 3
  36. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  37. 2025The Spotlight Resonance Method: Resolving the Alignment of Embedded Activationscited by 3
  38. 2025On the Unreasonable Effectiveness of Last-layer Retrainingcited by 3

50 most cited of 95 All 95 (JSON)