mlsanityData to 7 Oct 2026

The Persistence of Neural Collapse Despite Low-Rank Bias

arXiv 2410.23169 ↗First posted 30 Oct 2024cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Neural collapse (NC) and its multi-layer variant, deep neural collapse (DNC), describe a structured geometry that occurs in the features and weights of trained deep networks. Recent theoretical work by Sukenik et al. using a deep unconstrained feature model (UFM) suggests that DNC is suboptimal under mean squared error (MSE) loss. They heuristically argue that this is due to low-rank bias induced by L2 regularization. In this work, we extend this result to deep UFMs trained with cross-entropy loss, showing that high-rank structures, including DNC, are not generally optimal. We characterize the associated low-rank bias, proving a fixed bound on the number of non-negligible singular values at global minima as network depth increases. We further analyze the loss surface, demonstrating that DNC is more prevalent in the landscape than other critical configurations, which we argue explains its frequent empirical appearance. Our results are validated through experiments in deep UFMs and deep neural networks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 72 of its 86 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

72 of 86 references.

References · 72

  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  3. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  4. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  5. 2014The Loss Surfaces of Multilayer Networkscited by 278
  6. 2016Deep Learning without Poor Local Minimacited by 233
  7. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  8. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  9. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
38 more
  1. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  4. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  5. 2020Neural collapse with unconstrained featurescited by 140
  6. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  7. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  8. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  9. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  10. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  11. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  12. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  13. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  14. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  15. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  16. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  17. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  18. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  19. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  20. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  21. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  22. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  23. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  24. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  25. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  26. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  27. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  28. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  29. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  30. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  31. 2022A Law of Data Separation in Deep Learningcited by 36
  32. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  33. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  34. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  35. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  36. 2021Unveiling the structure of wide flat minima in neural networkscited by 27
  37. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  38. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  2. 2025MaxSup: Overcoming Representation Collapse in Label Smoothingcited by 4
  3. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  4. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  5. 2026Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholdcited by 2
  6. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  7. 2026Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Featurescited by 1
  8. 2026The Implicit Bias of Logit Regularizationcited by 1
  9. 2026Optimizer choice matters for the emergence of Neural Collapsecited by 1
  10. 2025Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappingscited by 0
  11. 2026PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learningcited by 0
  12. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0