mlsanityData to 14 Oct 2026

An Unconstrained Layer-Peeled Perspective on Neural Collapse

arXiv 2110.02796 ↗First posted 6 Oct 2021cs.LG stat.MLcited by 87

Not on the map. Why?

Abstract (arXiv)

Neural collapse is a highly symmetric geometric pattern of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

44 of 67 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2018Deep Learning for Classical Japanese Literaturecited by 422
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  10. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  11. 2017PDE-Net: Learning PDEs from Datacited by 311
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
32 more
  1. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  3. 2017Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanicscited by 214
  4. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  7. 2020Neural collapse with unconstrained featurescited by 142
  8. 2016Matrix Completion has No Spurious Local Minimumcited by 125
  9. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  10. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  11. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  12. 2018Convergence of Gradient Descent on Separable Datacited by 110
  13. 2016A Geometric Analysis of Phase Retrievalcited by 92
  14. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
  15. 2020Gradient descent follows the regularization path for general lossescited by 59
  16. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  17. 2020Neural Collapse with Cross-Entropy Losscited by 52
  18. 2015Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picturecited by 49
  19. 2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 42
  20. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  21. 2021Adversarial Training Helps Transfer Learning via Better Representationscited by 36
  22. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  23. 2020When Does Preconditioning Help or Hurt Generalization?cited by 31
  24. 2020From Symmetry to Geometry: Tractable Nonconvex Problemscited by 29
  25. 2015When Are Nonconvex Problems Not Scary?cited by 28
  26. 2020Representation via Representations: Domain Generalization via Adversarially Learned Invariant Representationscited by 26
  27. 2015Global Optimality in Tensor Factorization, Deep Learning, and Beyondcited by 22
  28. 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
  29. 2021Understanding Dynamics of Nonlinear Representation Learning and Its Applicationcited by 11
  30. 2019Short-and-Sparse Deconvolution -- A Geometric Approachcited by 9
  31. 2019Analysis of the Optimization Landscapes for Overcomplete Representation Learningcited by 7
  32. 2020To Each Optimizer a Norm, To Each Norm its Generalizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 19; 2023: 26; 2024: 22; 2025: 12; 2026: 8.

87 papers.

Sort:

By area:

  1. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  2. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  3. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  4. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  5. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  6. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  7. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  8. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  9. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  10. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  11. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  12. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
38 more
  1. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  2. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  3. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  4. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  5. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  6. 2022Perturbation Analysis of Neural Collapsecited by 35
  7. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  8. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  9. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  10. 2022Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networkscited by 23
  11. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  12. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  13. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  14. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  15. 2023HappyMap: A Generalized Multi-calibration Methodcited by 18
  16. 2023Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networkscited by 17
  17. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  18. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  19. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  20. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  21. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  22. 2022FIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced Datacited by 13
  23. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  24. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  25. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  26. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  27. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 11
  28. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  29. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  30. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  31. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  32. 2024Formation of Representations in Neural Networkscited by 10
  33. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  34. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  35. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  36. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 9
  37. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  38. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8

50 most cited of 87 All 87 (JSON)