# Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers

arXiv: https://arxiv.org/abs/2505.15239 · HTML: https://mlsanity.com/paper/2505.15239/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 70 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Balanced Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2207.09052/) (2022)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022)
- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [Limitations of Neural Collapse for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2202.08384/) (2022)
- [No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier](https://mlsanity.com/paper/2303.10058/) (2023)
- [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020)
- [Understanding Imbalanced Semantic Segmentation Through Neural Collapse](https://mlsanity.com/paper/2301.01100/) (2023)
- [Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model](https://mlsanity.com/paper/2305.13165/) (2023)
- [Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning](https://mlsanity.com/paper/2204.07596/) (2022)
- [Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data](https://mlsanity.com/paper/2301.00437/) (2023)
- [Generalized Neural Collapse for a Large Number of Classes](https://mlsanity.com/paper/2310.05351/) (2023)
- [Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data](https://mlsanity.com/paper/2309.09725/) (2023)
- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [A Law of Data Separation in Deep Learning](https://mlsanity.com/paper/2210.17020/) (2022)
- [Perturbation Analysis of Neural Collapse](https://mlsanity.com/paper/2210.16658/) (2022)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [The Tunnel Effect: Building Data Representations in Deep Neural Networks](https://mlsanity.com/paper/2305.19753/) (2023)
- [Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination](https://mlsanity.com/paper/2311.02960/) (2023)

## Built on by (most cited first, 10 of 10)

- [Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics](https://mlsanity.com/paper/2512.04006/) (2025)
- [Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold](https://mlsanity.com/paper/2604.00230/) (2026)
- [Structure Before Collapse: Transient semantic geometry in next-token prediction](https://mlsanity.com/paper/2606.26749/) (2026)
- [Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension](https://mlsanity.com/paper/2510.01105/) (2025)
- [Heads collapse, features stay: Why Replay needs big buffers](https://mlsanity.com/paper/2512.07400/) (2025)
- [Deep Neural Regression Collapse](https://mlsanity.com/paper/2603.23805/) (2026)
- [The Geometry of Projection Heads: Conditioning, Invariance, and Collapse](https://mlsanity.com/paper/2605.17180/) (2026)
- [Generalization in Nonlinear Least Squares via Learned Feature Geometry](https://mlsanity.com/paper/2606.08799/) (2026)
- [The Implicit Bias of Depth: From Neural Collapse to Softmax Codes](https://mlsanity.com/paper/2605.23087/) (2026)
- [CAENL: Collapse-Based Active Neural Learning with Objective-Aligned Representation Control](https://mlsanity.com/paper/2610.17509/) (2026)

## Abstract (arXiv)

The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spurred a line of theoretical research aimed at its understanding. However, existing work focuses on data-agnostic models or, when data structure is taken into account, it remains limited to multi-layer perceptrons. Our paper fills both these gaps by analyzing modern architectures in a data-aware regime: we prove that global optima of deep regularized transformers and residual networks (ResNets) with LayerNorm trained with cross entropy or mean squared error loss are approximately collapsed, and the approximation gets tighter as the depth grows. More generally, we formally reduce any end-to-end large-depth ResNet or transformer training into an equivalent unconstrained features model, thus justifying its wide use in the literature even beyond data-agnostic settings. Our theoretical results are supported by experiments on computer vision and language datasets showing that, as the depth grows, neural collapse indeed becomes more prominent.