mlsanityData to 7 Oct 2026

Perturbation Analysis of Neural Collapse

arXiv 2210.16658 ↗First posted 29 Oct 2022cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks for classification often includes minimizing the training loss beyond the zero training error point. In this phase of training, a "neural collapse" behavior has been observed: the variability of features (outputs of the penultimate layer) of within-class samples decreases and the mean features of different classes approach a certain tight frame structure. Recent works analyze this behavior via idealized unconstrained features models where all the minimizers exhibit exact collapse. However, with practical networks and datasets, the features typically do not reach exact collapse, e.g., because deep layers cannot arbitrarily modify intermediate features that are far from being collapsed. In this paper, we propose a richer model that can capture this phenomenon by forcing the features to stay in the vicinity of a predefined features matrix (e.g., intermediate features). We explore the model in the small vicinity case via perturbation analysis and establish results that cannot be obtained by the previously studied models. For example, we prove reduction in the within-class variability of the optimized features compared to the predefined input features (via analyzing gradient flow on the "central-path" with minimal assumptions), analyze the minimizers in the near-collapse regime, and provide insights on the effect of regularization hyperparameters on the closeness to collapse. We support our theory with experiments in practical deep learning settings.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

25 of 33 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2019Deep Equilibrium Modelscited by 603
  6. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  7. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  8. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  9. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  10. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  11. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  12. 2020Neural collapse with unconstrained featurescited by 140
13 more
  1. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  2. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  3. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  4. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  5. 2018Does data interpolation contradict statistical optimality?cited by 95
  6. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  7. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  8. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  9. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  10. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  11. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  12. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  13. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 15; 2025: 4; 2026: 7.

34 papers.

Sort:

By area:

  1. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  2. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  3. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  4. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  5. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  6. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  7. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  8. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  9. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  10. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  11. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  12. 2023Neural (Tangent Kernel) Collapsecited by 18
22 more
  1. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  2. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  3. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  4. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  5. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  6. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  7. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 9
  8. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  9. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 8
  10. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  11. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  12. 2024Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)cited by 2
  13. 2024SGW-based Multi-Task Learning in Vision Taskscited by 2
  14. 2026Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholdcited by 2
  15. 2025Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregationcited by 0
  16. 2025Enhancing Conformal Prediction via Class Similaritycited by 0
  17. 2026How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapsecited by 0
  18. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  19. 2026The role of class encoding in neural collapsecited by 0
  20. 2026Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learningcited by 0
  21. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0
  22. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0