# Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

arXiv: https://arxiv.org/abs/2301.00437 · HTML: https://mlsanity.com/paper/2301.00437/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 1 Jan 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?](https://mlsanity.com/paper/2203.09081/) (2022)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [Does data interpolation contradict statistical optimality?](https://mlsanity.com/paper/1806.09471/) (2018)
- [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Neural Collapse with Cross-Entropy Loss](https://mlsanity.com/paper/2012.08465/) (2020)
- [Small nonlinearities in activation functions create bad local minima in neural networks](https://mlsanity.com/paper/1802.03487/) (2018)

## Built on by (most cited first, 30 of 40)

- [Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model](https://mlsanity.com/paper/2305.13165/) (2023)
- [Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data](https://mlsanity.com/paper/2309.09725/) (2023)
- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [Average gradient outer product as a mechanism for deep neural collapse](https://mlsanity.com/paper/2402.13728/) (2024)
- [The Prevalence of Neural Collapse in Neural Multivariate Regression](https://mlsanity.com/paper/2409.04180/) (2024)
- [Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model](https://mlsanity.com/paper/2401.02058/) (2024)
- [Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?](https://mlsanity.com/paper/2405.14468/) (2024)
- [Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse](https://mlsanity.com/paper/2410.04887/) (2024)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)
- [Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse](https://mlsanity.com/paper/2405.05587/) (2024)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Progressive Feedforward Collapse of ResNet Training](https://mlsanity.com/paper/2405.00985/) (2024)
- [Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection](https://mlsanity.com/paper/2405.17816/) (2024)
- [Cross Entropy versus Label Smoothing: A Neural Collapse Perspective](https://mlsanity.com/paper/2402.03979/) (2024)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data](https://mlsanity.com/paper/2401.08977/) (2024)
- [Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data](https://mlsanity.com/paper/2409.01832/) (2024)
- [Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers](https://mlsanity.com/paper/2505.15239/) (2025)
- [Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking](https://mlsanity.com/paper/2509.17738/) (2025)
- [Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model](https://mlsanity.com/paper/2506.05801/) (2025)
- [Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching](https://mlsanity.com/paper/2306.07960/) (2023)
- [Can Kernel Methods Explain How the Data Affects Neural Collapse?](https://mlsanity.com/paper/2406.02105/) (2024)
- [The Fair Language Model Paradox](https://mlsanity.com/paper/2410.11985/) (2024)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective](https://mlsanity.com/paper/2410.23391/) (2024)
- [On the Unreasonable Effectiveness of Last-layer Retraining](https://mlsanity.com/paper/2512.01766/) (2025)
- [MASIL: Towards Maximum Separable Class Representation for Few Shot Class Incremental Learning](https://mlsanity.com/paper/2304.05362/) (2023)
- [SegDA: Maximum Separable Segment Mask with Pseudo Labels for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2308.05851/) (2023)
- [Neural Collapse Inspired Knowledge Distillation](https://mlsanity.com/paper/2412.11788/) (2024)
- [Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model](https://mlsanity.com/paper/2505.09308/) (2025)

## Abstract (arXiv)

Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing. Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is known as Neural Collapse (NC). Recent papers have theoretically shown that NC emerges in the global minimizers of training problems with the simplified "unconstrained feature model". In this context, we take a step further and prove the NC occurrences in deep linear networks for the popular mean squared error (MSE) and cross entropy (CE) losses, showing that global solutions exhibit NC properties across the linear layers. Furthermore, we extend our study to imbalanced data for MSE loss and present the first geometric analysis of NC under bias-free setting. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of orthogonal vectors, whose lengths depend on the amount of data in their corresponding classes. Finally, we empirically validate our theoretical analyses on synthetic and practical network architectures with both balanced and imbalanced scenarios.