mlsanityData to 14 Oct 2026

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

arXiv 2301.00437 ↗First posted 1 Jan 2023cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing. Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is known as Neural Collapse (NC). Recent papers have theoretically shown that NC emerges in the global minimizers of training problems with the simplified "unconstrained feature model". In this context, we take a step further and prove the NC occurrences in deep linear networks for the popular mean squared error (MSE) and cross entropy (CE) losses, showing that global solutions exhibit NC properties across the linear layers. Furthermore, we extend our study to imbalanced data for MSE loss and present the first geometric analysis of NC under bias-free setting. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of orthogonal vectors, whose lengths depend on the amount of data in their corresponding classes. Finally, we empirically validate our theoretical analyses on synthetic and practical network architectures with both balanced and imbalanced scenarios.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 17 of its 42 references that are in the map. First posted on arXiv 1 Jan 2023.

Builds on

17 of 42 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2016An overview of gradient descent optimization algorithmscited by 1,033
  3. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  4. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  6. 2016Deep Learning without Poor Local Minimacited by 234
  7. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  8. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  9. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  10. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  11. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  12. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
5 more
  1. 2018Does data interpolation contradict statistical optimality?cited by 95
  2. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  3. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  4. 2020Neural Collapse with Cross-Entropy Losscited by 52
  5. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 18; 2025: 10; 2026: 6.

40 papers.

Sort:

By area:

  1. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  2. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  3. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  4. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  5. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  6. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  7. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  8. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  9. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  10. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  11. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  12. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
28 more
  1. 2024Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detectioncited by 12
  2. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  3. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  4. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  5. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  6. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  7. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  8. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  9. 2023Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batchingcited by 5
  10. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  11. 2024The Fair Language Model Paradoxcited by 4
  12. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  13. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  14. 2025On the Unreasonable Effectiveness of Last-layer Retrainingcited by 3
  15. 2023MASIL: Towards Maximum Separable Class Representation for Few Shot Class Incremental Learningcited by 2
  16. 2023SegDA: Maximum Separable Segment Mask with Pseudo Labels for Domain Adaptive Semantic Segmentationcited by 2
  17. 2024Neural Collapse Inspired Knowledge Distillationcited by 2
  18. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  19. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  20. 2026Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmeticcited by 2
  21. 2025Heads collapse, features stay: Why Replay needs big bufferscited by 1
  22. 2026Equilibrium contrastive learning for imbalanced image classificationcited by 1
  23. 2025Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detectioncited by 0
  24. 2025Exploring "Many in Few" and "Few in Many" Properties in Long-Tailed, Highly-Imbalanced IC Defect Classificationcited by 0
  25. 2026How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapsecited by 0
  26. 2026Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasetscited by 0
  27. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  28. 2026Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imagingcited by 0