# Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model

arXiv: https://arxiv.org/abs/2305.13165 · HTML: https://mlsanity.com/paper/2305.13165/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 31)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier](https://mlsanity.com/paper/2303.10058/) (2023)
- [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Understanding Imbalanced Semantic Segmentation Through Neural Collapse](https://mlsanity.com/paper/2301.01100/) (2023)
- [Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning](https://mlsanity.com/paper/2204.08735/) (2022)
- [Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data](https://mlsanity.com/paper/2301.00437/) (2023)
- [Explicit regularization and implicit bias in deep network classifiers trained with the square loss](https://mlsanity.com/paper/2101.00072/) (2020)
- [A Law of Data Separation in Deep Learning](https://mlsanity.com/paper/2210.17020/) (2022)
- [Perturbation Analysis of Neural Collapse](https://mlsanity.com/paper/2210.16658/) (2022)
- [Separation and Concentration in Deep Networks](https://mlsanity.com/paper/2012.10424/) (2020)
- [The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks](https://mlsanity.com/paper/2210.03820/) (2022)
- [Understanding and Improving Transfer Learning of Deep Models via Neural Collapse](https://mlsanity.com/paper/2212.12206/) (2022)
- [Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks](https://mlsanity.com/paper/2209.08378/) (2022)
- [Nearest Class-Center Simplification through Intermediate Layers](https://mlsanity.com/paper/2201.08924/) (2022)
- [Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap](https://mlsanity.com/paper/2303.06484/) (2023)
- [Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity](https://mlsanity.com/paper/2303.05689/) (2023)
- [On the Implicit Bias Towards Minimal Depth of Deep Neural Networks](https://mlsanity.com/paper/2202.09028/) (2022)

## Built on by (most cited first, 30 of 43)

- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [Average gradient outer product as a mechanism for deep neural collapse](https://mlsanity.com/paper/2402.13728/) (2024)
- [The Prevalence of Neural Collapse in Neural Multivariate Regression](https://mlsanity.com/paper/2409.04180/) (2024)
- [Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model](https://mlsanity.com/paper/2401.02058/) (2024)
- [Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?](https://mlsanity.com/paper/2405.14468/) (2024)
- [Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse](https://mlsanity.com/paper/2410.04887/) (2024)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Progressive Feedforward Collapse of ResNet Training](https://mlsanity.com/paper/2405.00985/) (2024)
- [Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networks](https://mlsanity.com/paper/2402.03991/) (2024)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [Transformer Block Coupling and its Correlation with Generalization in LLMs](https://mlsanity.com/paper/2407.07810/) (2024)
- [Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data](https://mlsanity.com/paper/2409.01832/) (2024)
- [Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers](https://mlsanity.com/paper/2505.15239/) (2025)
- [Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay](https://mlsanity.com/paper/2309.04644/) (2023)
- [Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)](https://mlsanity.com/paper/2502.21009/) (2025)
- [Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking](https://mlsanity.com/paper/2509.17738/) (2025)
- [NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering](https://mlsanity.com/paper/2510.05635/) (2025)
- [Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model](https://mlsanity.com/paper/2506.05801/) (2025)
- [Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching](https://mlsanity.com/paper/2306.07960/) (2023)
- [Can Kernel Methods Explain How the Data Affects Neural Collapse?](https://mlsanity.com/paper/2406.02105/) (2024)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [A mathematical theory for understanding when abstract representations emerge in neural networks](https://mlsanity.com/paper/2510.09816/) (2025)
- [Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization](https://mlsanity.com/paper/2506.01562/) (2025)
- [Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations](https://mlsanity.com/paper/2505.08348/) (2025)
- [Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model](https://mlsanity.com/paper/2505.09308/) (2025)
- [Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data](https://mlsanity.com/paper/2510.21078/) (2025)
- [Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics](https://mlsanity.com/paper/2512.04006/) (2025)
- [Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Features](https://mlsanity.com/paper/2601.12011/) (2026)
- [Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold](https://mlsanity.com/paper/2604.00230/) (2026)
- [Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets](https://mlsanity.com/paper/2405.17573/) (2024)

## Abstract (arXiv)

Neural collapse (NC) refers to the surprising structure of the last layer of deep neural networks in the terminal phase of gradient descent training. Recently, an increasing amount of experimental evidence has pointed to the propagation of NC to earlier layers of neural networks. However, while the NC in the last layer is well studied theoretically, much less is known about its multi-layered counterpart - deep neural collapse (DNC). In particular, existing work focuses either on linear layers or only on the last two layers at the price of an extra assumption. Our paper fills this gap by generalizing the established analytical framework for NC - the unconstrained features model - to multiple non-linear layers. Our key technical contribution is to show that, in a deep unconstrained features model, the unique global optimum for binary classification exhibits all the properties typical of DNC. This explains the existing experimental evidence of DNC. We also empirically show that (i) by optimizing deep unconstrained features models via gradient descent, the resulting solution agrees well with our theory, and (ii) trained networks recover the unconstrained features suitable for the occurrence of DNC, thus supporting the validity of this modeling principle.