mlsanityData to 7 Oct 2026

Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data

arXiv 2510.21078 ↗First posted 24 Oct 2025cs.LG math.OCcited by 2

Not on the map. Why?

Abstract (arXiv)

Among many mysteries behind the success of deep networks lies the exceptional discriminative power of their learned representations as manifested by the intriguing Neural Collapse (NC) phenomenon, where simple feature structures emerge at the last layer of a trained neural network. Prior works on the theoretical understandings of NC have focused on analyzing the optimization landscape of matrix-factorization-like problems by considering the last-layer features as unconstrained free optimization variables and showing that their global minima exhibit NC. In this paper, we show that gradient flow on a two-layer ReLU network for classifying orthogonally separable data provably exhibits NC, thereby advancing prior results in two ways: First, we relax the assumption of unconstrained features, showing the effect of data structure and nonlinear activations on NC characterizations. Second, we reveal the role of the implicit bias of the training dynamics in facilitating the emergence of NC.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 57 of its 75 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

57 of 75 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
38 more
  1. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  6. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  7. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  8. 2020Directional convergence and alignment in deep learningcited by 172
  9. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  10. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  11. 2020Neural collapse with unconstrained featurescited by 140
  12. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  13. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  14. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  15. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  16. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  17. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  18. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  19. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  20. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  21. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  22. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  23. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  24. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  25. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  26. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  27. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  28. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  29. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  30. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficientlycited by 35
  31. 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
  32. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  33. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  34. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  35. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 19
  36. 2024Early alignment in two-layer networks training is a two-edged swordcited by 17
  37. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  38. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networkscited by 3
  2. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0