mlsanityData to 14 Oct 2026

Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers

arXiv 2505.15239 ↗First posted 21 May 2025cs.LG cs.AI stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spurred a line of theoretical research aimed at its understanding. However, existing work focuses on data-agnostic models or, when data structure is taken into account, it remains limited to multi-layer perceptrons. Our paper fills both these gaps by analyzing modern architectures in a data-aware regime: we prove that global optima of deep regularized transformers and residual networks (ResNets) with LayerNorm trained with cross entropy or mean squared error loss are approximately collapsed, and the approximation gets tighter as the depth grows. More generally, we formally reduce any end-to-end large-depth ResNet or transformer training into an equivalent unconstrained features model, thus justifying its wide use in the literature even beyond data-agnostic settings. Our theoretical results are supported by experiments on computer vision and language datasets showing that, as the depth grows, neural collapse indeed becomes more prominent.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 54 of its 70 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

54 of 70 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  4. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  5. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  8. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 144
  9. 2020Neural collapse with unconstrained featurescited by 142
  10. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  11. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  12. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
38 more
  1. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  2. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  3. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  4. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  5. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  6. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  7. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  8. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  9. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  10. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  11. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  12. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  13. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  14. 2022A Law of Data Separation in Deep Learningcited by 36
  15. 2022Perturbation Analysis of Neural Collapsecited by 35
  16. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  17. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  18. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 32
  19. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  20. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  21. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  22. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  23. 2022Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networkscited by 23
  24. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  25. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  26. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  27. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  28. 2023Neural (Tangent Kernel) Collapsecited by 18
  29. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  30. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  31. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  32. 2023Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severitycited by 13
  33. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  34. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  35. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  36. 2024Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detectioncited by 12
  37. 2024EPA: Neural Collapse Inspired Robust Out-of-Distribution Detectorcited by 10
  38. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  2. 2026Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholdcited by 2
  3. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  4. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  5. 2025Heads collapse, features stay: Why Replay needs big bufferscited by 1
  6. 2026Deep Neural Regression Collapsecited by 1
  7. 2026The Geometry of Projection Heads: Conditioning, Invariance, and Collapsecited by 1
  8. 2026Generalization in Nonlinear Least Squares via Learned Feature Geometrycited by 1
  9. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  10. 2026CAENL: Collapse-Based Active Neural Learning with Objective-Aligned Representation Controlcited by 0