mlsanityData to 7 Oct 2026

Neural Collapse in Multi-label Learning with Pick-all-label Loss

arXiv 2310.15903 ↗First posted 24 Oct 2023cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

We study deep neural networks for the multi-label classification (MLab) task through the lens of neural collapse (NC). Previous works have been restricted to the multi-class classification setting and discovered a prevalent NC phenomenon comprising of the following properties for the last-layer features: (i) the variability of features within every class collapses to zero, (ii) the set of feature means form an equi-angular tight frame (ETF), and (iii) the last layer classifiers collapse to the feature mean upon some scaling. We generalize the study to multi-label learning, and prove for the first time that a generalized NC phenomenon holds with the "pick-all-label" formulation, which we term as MLab NC. While the ETF geometry remains consistent for features with a single label, multi-label scenarios introduce a unique combinatorial aspect we term the "tag-wise average" property, where the means of features with multiple labels are the scaled averages of means for single-label instances. Theoretically, under proper assumptions on the features, we establish that the only global optimizer of the pick-all-label cross-entropy loss satisfy the multi-label NC. In practice, we demonstrate that our findings can lead to better test performance with more efficient training techniques for MLab learning.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 57 of its 89 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

57 of 89 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  9. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  10. 2017How to Escape Saddle Points Efficientlycited by 284
  11. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  12. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
38 more
  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  3. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  4. 2020Neural collapse with unconstrained featurescited by 140
  5. 2020General Multi-label Image Classification with Transformerscited by 126
  6. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  7. 2019Learning with Fenchel-Young Lossescited by 117
  8. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  9. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  10. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  11. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  12. 2020The Emerging Trends of Multi-Label Learningcited by 99
  13. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  14. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  15. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  16. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  17. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  18. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  19. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 78
  20. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  21. 2021Dissecting Supervised Contrastive Learningcited by 64
  22. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  23. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  24. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  25. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  26. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  27. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  28. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  29. 2022A Law of Data Separation in Deep Learningcited by 36
  30. 2018Stochastic Negative Mining for Learning with Large Output Spacescited by 29
  31. 2020From Symmetry to Geometry: Tractable Nonconvex Problemscited by 28
  32. 2015When Are Nonconvex Problems Not Scary?cited by 27
  33. 2015Complete Dictionary Recovery over the Sphere II: Recovery by Riemannian Trust-region Methodcited by 24
  34. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  35. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  36. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  37. 2020Convex Calibrated Surrogates for the Multi-Label F-Measurecited by 18
  38. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 4.

14 papers.

Sort:

  1. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  2. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  3. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  4. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  5. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  6. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  7. 2025Neural Collapse-Inspired Multi-Label Federated Learning under Label-Distribution Skewcited by 2
  8. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  9. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  10. 2024ℓ1-norm rank-one symmetric matrix factorization has no spurious second-order stationary pointscited by 0
  11. 2026Bit-Identical Medical Deep Learning via Structured Orthogonal Initializationcited by 0
  12. 2026How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapsecited by 0
2 more
  1. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  2. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0