mlsanityData to 14 Oct 2026

Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data

arXiv 2309.09725 ↗First posted 18 Sep 2023stat.ML cs.LG math.OCcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the huge success of deep neural networks (DNNs) in various tasks of computer vision and text processing. Interestingly, these DNNs with massive number of parameters share similar structural properties on their feature representation and last-layer classifier at terminal phase of training (TPT). Specifically, if the training data are balanced (each class shares the same number of samples), it is observed that the feature vectors of samples from the same class converge to their corresponding in-class mean features and their pairwise angles are the same. This fascinating phenomenon is known as Neural Collapse (N C), first termed by Papyan, Han, and Donoho in 2019. Many recent works manage to theoretically explain this phenomenon by adopting so-called unconstrained feature model (UFM). In this paper, we study the extension of N C phenomenon to the imbalanced data under cross-entropy loss function in the context of unconstrained feature model. Our contribution is multi-fold compared with the state-of-the-art results: (a) we show that the feature vectors exhibit collapse phenomenon, i.e., the features within the same class collapse to the same mean vector; (b) the mean feature vectors no longer form an equiangular tight frame. Instead, their pairwise angles depend on the sample size; (c) we also precisely characterize the sharp threshold on which the minority collapse (the feature vectors of the minority groups collapse to one single vector) will take place; (d) finally, we argue that the effect of the imbalance in datasize diminishes as the sample size grows. Our results provide a complete picture of the N C under the cross-entropy loss for the imbalanced data. Numerical experiments confirm our theoretical analysis.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

30 of 39 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  6. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  9. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  10. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  11. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  12. 2020Neural collapse with unconstrained featurescited by 142
18 more
  1. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  2. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  3. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  4. 2018Does data interpolation contradict statistical optimality?cited by 95
  5. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  6. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  7. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  8. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  9. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  10. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  11. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  12. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  13. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  14. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  15. 2022Perturbation Analysis of Neural Collapsecited by 35
  16. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  17. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  18. 2023Neural (Tangent Kernel) Collapsecited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 15; 2025: 11; 2026: 11.

38 papers.

Sort:

By area:

  1. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  2. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  3. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  4. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  5. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  6. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  7. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  8. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  9. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 11
  10. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  11. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  12. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
26 more
  1. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
  2. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  3. 2025NEO: No-Optimization Test-Time Adaptation through Latent Re-Centeringcited by 8
  4. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  5. 2023Geometric structure of shallow neural networks and constructive mathcal L2 cost minimizationcited by 4
  6. 2024The Fair Language Model Paradoxcited by 4
  7. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3
  8. 2025On the Unreasonable Effectiveness of Last-layer Retrainingcited by 3
  9. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  10. 2025Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representationscited by 2
  11. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  12. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  13. 2026Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Featurescited by 2
  14. 2026Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholdcited by 2
  15. 2024Trojan Cleansing with Neural Collapsecited by 1
  16. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  17. 2025Heads collapse, features stay: Why Replay needs big bufferscited by 1
  18. 2026Safety-Efficacy Trade Off: Robustness against Data-Poisoningcited by 1
  19. 2026How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapsecited by 0
  20. 2026Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of Viewcited by 0
  21. 2026Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasetscited by 0
  22. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  23. 2026Ordinal Neural Collapse as a Representation Prior for Visual Navigationcited by 0
  24. 2026Neural Collapse Is Forbidden: Information Floors in Language Modelscited by 0
  25. 2026Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergencecited by 0
  26. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0