mlsanityData to 7 Oct 2026

Extended Unconstrained Features Model for Exploring Deep Neural Collapse

arXiv 2202.08087 ↗First posted 16 Feb 2022cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

The modern strategy for training deep neural networks for classification tasks includes optimizing the network's weights even after the training error vanishes to further push the training loss toward zero. Recently, a phenomenon termed "neural collapse" (NC) has been empirically observed in this training procedure. Specifically, it has been shown that the learned features (the output of the penultimate layer) of within-class samples converge to their mean, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer's weights. Recent papers have shown that minimizers with this structure emerge when optimizing a simplified "unconstrained features model" (UFM) with a regularized cross-entropy loss. In this paper, we further analyze and extend the UFM. First, we study the UFM for the regularized MSE loss, and show that the minimizers' features can have a more delicate structure than in the cross-entropy case. This affects also the structure of the weights. Then, we extend the UFM by adding another layer of weights as well as ReLU nonlinearity to the model and generalize our previous results. Finally, we empirically demonstrate the usefulness of our nonlinear extended UFM in modeling the NC phenomenon that occurs with practical networks.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

20 of 29 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  6. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  7. 2016Deep Learning without Poor Local Minimacited by 233
  8. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  9. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  10. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  11. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  12. 2020Neural collapse with unconstrained featurescited by 140
8 more
  1. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  2. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  3. 2018Does data interpolation contradict statistical optimality?cited by 95
  4. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  5. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  6. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  7. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  8. 2020Separation and Concentration in Deep Networkscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 29; 2024: 24; 2025: 20; 2026: 12.

101 papers.

Sort:

By area:

  1. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  2. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  3. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  4. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  5. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  6. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  7. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  8. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  9. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  10. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  11. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  12. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
38 more
  1. 2023NECO: NEural Collapse Based Out-of-distribution detectioncited by 41
  2. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  3. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  4. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  5. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  6. 2022Perturbation Analysis of Neural Collapsecited by 34
  7. 2023Reverse Engineering Self-Supervised Learningcited by 33
  8. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  9. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  10. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  11. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  12. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  13. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  14. 2023Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networkscited by 20
  15. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  16. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  17. 2023Neural (Tangent Kernel) Collapsecited by 18
  18. 2024CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignmentcited by 16
  19. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  20. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  21. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  22. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  23. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  24. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  25. 2023Unsupervised Manifold Linearizing and Clusteringcited by 12
  26. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  27. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  28. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  29. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  30. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  31. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  32. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  33. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  34. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 9
  35. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  36. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 8
  37. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
  38. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8

50 most cited of 101 All 101 (JSON)