# Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data

arXiv: https://arxiv.org/abs/2309.09725 · HTML: https://mlsanity.com/paper/2309.09725/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [Does data interpolation contradict statistical optimality?](https://mlsanity.com/paper/1806.09471/) (2018)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [Revealing the Structure of Deep Neural Networks via Convex Duality](https://mlsanity.com/paper/2002.09773/) (2020)
- [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data](https://mlsanity.com/paper/2301.00437/) (2023)
- [Explicit regularization and implicit bias in deep network classifiers trained with the square loss](https://mlsanity.com/paper/2101.00072/) (2020)
- [Perturbation Analysis of Neural Collapse](https://mlsanity.com/paper/2210.16658/) (2022)
- [Understanding and Improving Transfer Learning of Deep Models via Neural Collapse](https://mlsanity.com/paper/2212.12206/) (2022)
- [On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data](https://mlsanity.com/paper/2303.07608/) (2023)
- [Neural (Tangent Kernel) Collapse](https://mlsanity.com/paper/2305.16427/) (2023)

## Built on by (most cited first, 30 of 38)

- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [Average gradient outer product as a mechanism for deep neural collapse](https://mlsanity.com/paper/2402.13728/) (2024)
- [The Prevalence of Neural Collapse in Neural Multivariate Regression](https://mlsanity.com/paper/2409.04180/) (2024)
- [Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model](https://mlsanity.com/paper/2401.02058/) (2024)
- [Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?](https://mlsanity.com/paper/2405.14468/) (2024)
- [Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse](https://mlsanity.com/paper/2410.04887/) (2024)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Cross Entropy versus Label Smoothing: A Neural Collapse Perspective](https://mlsanity.com/paper/2402.03979/) (2024)
- [Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networks](https://mlsanity.com/paper/2402.03991/) (2024)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data](https://mlsanity.com/paper/2409.01832/) (2024)
- [Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers](https://mlsanity.com/paper/2505.15239/) (2025)
- [Control-oriented Clustering of Visual Latent Representation](https://mlsanity.com/paper/2410.05063/) (2024)
- [Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking](https://mlsanity.com/paper/2509.17738/) (2025)
- [NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering](https://mlsanity.com/paper/2510.05635/) (2025)
- [Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model](https://mlsanity.com/paper/2506.05801/) (2025)
- [Geometric structure of shallow neural networks and constructive mathcal L2 cost minimization](https://mlsanity.com/paper/2309.10370/) (2023)
- [The Fair Language Model Paradox](https://mlsanity.com/paper/2410.11985/) (2024)
- [Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization](https://mlsanity.com/paper/2506.01562/) (2025)
- [On the Unreasonable Effectiveness of Last-layer Retraining](https://mlsanity.com/paper/2512.01766/) (2025)
- [Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement](https://mlsanity.com/paper/2410.04264/) (2024)
- [Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations](https://mlsanity.com/paper/2505.08348/) (2025)
- [Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model](https://mlsanity.com/paper/2505.09308/) (2025)
- [Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics](https://mlsanity.com/paper/2512.04006/) (2025)
- [Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Features](https://mlsanity.com/paper/2601.12011/) (2026)
- [Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold](https://mlsanity.com/paper/2604.00230/) (2026)
- [Trojan Cleansing with Neural Collapse](https://mlsanity.com/paper/2411.12914/) (2024)
- [Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension](https://mlsanity.com/paper/2510.01105/) (2025)
- [Heads collapse, features stay: Why Replay needs big buffers](https://mlsanity.com/paper/2512.07400/) (2025)
- [Safety-Efficacy Trade Off: Robustness against Data-Poisoning](https://mlsanity.com/paper/2602.00822/) (2026)

## Abstract (arXiv)

Recent years have witnessed the huge success of deep neural networks (DNNs) in various tasks of computer vision and text processing. Interestingly, these DNNs with massive number of parameters share similar structural properties on their feature representation and last-layer classifier at terminal phase of training (TPT). Specifically, if the training data are balanced (each class shares the same number of samples), it is observed that the feature vectors of samples from the same class converge to their corresponding in-class mean features and their pairwise angles are the same. This fascinating phenomenon is known as Neural Collapse (N C), first termed by Papyan, Han, and Donoho in 2019. Many recent works manage to theoretically explain this phenomenon by adopting so-called unconstrained feature model (UFM). In this paper, we study the extension of N C phenomenon to the imbalanced data under cross-entropy loss function in the context of unconstrained feature model. Our contribution is multi-fold compared with the state-of-the-art results: (a) we show that the feature vectors exhibit collapse phenomenon, i.e., the features within the same class collapse to the same mean vector; (b) the mean feature vectors no longer form an equiangular tight frame. Instead, their pairwise angles depend on the sample size; (c) we also precisely characterize the sharp threshold on which the minority collapse (the feature vectors of the minority groups collapse to one single vector) will take place; (d) finally, we argue that the effect of the imbalance in datasize diminishes as the sample size grows. Our results provide a complete picture of the N C under the cross-entropy loss for the imbalanced data. Numerical experiments confirm our theoretical analysis.