mlsanityData to 13 Oct 2026

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

arXiv 2405.14468 ↗First posted 23 May 2024cs.LG math.OC stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagation of neural collapse to earlier layers of DNNs -- a phenomenon called deep neural collapse (DNC). However, existing theoretical results are restricted to special cases: linear models, only two layers or binary classification. In contrast, we focus on non-linear models of arbitrary depth in multi-class classification and reveal a surprising qualitative shift. As soon as we go beyond two layers or two classes, DNC stops being optimal for the deep unconstrained features model (DUFM) -- the standard theoretical framework for the analysis of collapse. The main culprit is a low-rank bias of multi-layer regularization schemes: this bias leads to optimal solutions of even lower rank than the neural collapse. We support our theoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.

Key facts. Cited by 17 papers in the map (data through 13 Oct 2026). Builds on 50 of its 63 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

50 of 63 references.

References · 50

  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 716
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 652
  3. 2019Implicit Regularization in Deep Matrix Factorizationcited by 404
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  5. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  8. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 152
  9. 2020Neural collapse with unconstrained featurescited by 142
  10. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  11. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  12. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
38 more
  1. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  2. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  3. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  4. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  5. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  6. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  7. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  8. 2022Rank Diminishing in Deep Neural Networkscited by 47
  9. 2021Implicit Regularization in Tensor Factorizationcited by 44
  10. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  11. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  12. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  13. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  14. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  15. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  16. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  17. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  18. 2022A Law of Data Separation in Deep Learningcited by 36
  19. 2022Perturbation Analysis of Neural Collapsecited by 35
  20. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  21. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  22. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  23. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  24. 2022Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networkscited by 23
  25. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  26. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  27. 2023Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networkscited by 20
  28. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  29. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  30. 2023Neural (Tangent Kernel) Collapsecited by 18
  31. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  32. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 16
  33. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  34. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  35. 2024EPA: Neural Collapse Inspired Robust Out-of-Distribution Detectorcited by 10
  36. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  37. 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
  38. 2024Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapsecited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  2. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  3. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  4. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  5. 2024Learning to Compress: Local Rank and Information Compression in Deep Neural Networkscited by 6
  6. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  7. 2025MaxSup: Overcoming Representation Collapse in Label Smoothingcited by 4
  8. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3
  9. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  10. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  11. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  12. 2026The Implicit Bias of Logit Regularizationcited by 1
5 more
  1. 2026Neural Collapse by Design: Learning Class Prototypes on the Hyperspherecited by 0
  2. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  3. 2026Feature Lottery? A Bifurcation Theory of Concept Emergencecited by 0
  4. 2026The role of class encoding in neural collapsecited by 0
  5. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMscited by 0