mlsanityData to 14 Oct 2026

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

arXiv 2202.08384 ↗First posted 17 Feb 2022cs.LG cs.CV stat.MLcited by 63

Not on the map. Why?

Abstract (arXiv)

The recent work of Papyan, Han, & Donoho (2020) presented an intriguing "Neural Collapse" phenomenon, showing a structural property of interpolating classifiers in the late stage of training. This opened a rich area of exploration studying this phenomenon. Our motivation is to study the upper limits of this research program: How far will understanding Neural Collapse take us in understanding deep learning? First, we investigate its role in generalization. We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property). We find that while Neural Collapse often occurs on the train set, it does not occur on the test set. We thus conclude that Neural Collapse is primarily an optimization phenomenon, with as-yet-unclear connections to generalization. Second, we investigate the role of Neural Collapse in feature learning. We show simple, realistic experiments where training longer leads to worse last-layer features, as measured by transfer-performance on a downstream task. This suggests that neural collapse is not always desirable for representation learning, as previously claimed. Finally, we give preliminary evidence of a "cascading collapse" phenomenon, wherein some form of Neural Collapse occurs not only for the last layer, but in earlier layers as well. We hope our work encourages the community to continue the rich line of Neural Collapse research, while also considering its inherent limitations.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 15 of its 31 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

15 of 31 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  5. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  8. 2020Neural collapse with unconstrained featurescited by 142
  9. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  10. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  11. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  12. 2020Neural Collapse with Cross-Entropy Losscited by 52
3 more
  1. 2019One Epoch Is All You Needcited by 41
  2. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  3. 2021Distribution of Classification Margins: Are All Data Equal?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 16; 2024: 14; 2025: 11; 2026: 6.

63 papers.

Sort:

By area:

  1. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  4. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  5. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  6. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  7. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  8. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  9. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  10. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  11. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  12. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
38 more
  1. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  2. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  3. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  4. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  5. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  6. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  7. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  8. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  9. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  10. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  11. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  12. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  13. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  14. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  15. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  16. 2024Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detectioncited by 12
  17. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  18. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  19. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  20. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  21. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 9
  22. 2025Double descent in quantum kernel methodscited by 9
  23. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  24. 2023Towards Demystifying the Generalization Behaviors When Neural Collapse Emergescited by 7
  25. 2022Efficient Maximal Coding Rate Reduction by Variational Formscited by 6
  26. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  27. 2022Class Based Thresholding in Early Exit Semantic Segmentation Networkscited by 5
  28. 2023Towards understanding neural collapse in supervised contrastive learning with the information bottleneck methodcited by 5
  29. 2023Deep neural networks architectures from the perspective of manifold learningcited by 5
  30. 2024Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Featurescited by 5
  31. 2022Gradient Descent for Low-Rank Functionscited by 3
  32. 2023On the Dynamics Under the Unhinged Loss and Beyondcited by 3
  33. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  34. 2025On Entity Identification in Language Modelscited by 3
  35. 2025On the Unreasonable Effectiveness of Last-layer Retrainingcited by 3
  36. 2022Memorization-Dilation: Modeling Neural Collapse Under Label Noisecited by 2
  37. 2023An Investigation of Representation and Allocation Harms in Contrastive Learningcited by 2
  38. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2

50 most cited of 63 All 63 (JSON)