mlsanityData to 7 Oct 2026

Quantifying the Variability Collapse of Neural Networks

arXiv 2306.03440 ↗First posted 6 Jun 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recent studies empirically demonstrate the positive relationship between the transferability of neural networks and the within-class variation of the last layer features. The recently discovered Neural Collapse (NC) phenomenon provides a new perspective of understanding such last layer geometry of neural networks. In this paper, we propose a novel metric, named Variability Collapse Index (VCI), to quantify the variability collapse phenomenon in the NC paradigm. The VCI metric is well-motivated and intrinsically related to the linear probing loss on the last layer features. Moreover, it enjoys desired theoretical and empirical properties, including invariance under invertible linear transformations and numerical stability, that distinguishes it from previous metrics. Our experiments verify that VCI is indicative of the variability collapse and the transferability of pretrained neural networks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

49 of 62 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  12. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
37 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  3. 2018Do Better ImageNet Models Transfer Better?cited by 560
  4. 2018A Survey on Deep Transfer Learningcited by 417
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  6. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  7. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  8. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  9. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 143
  10. 2020Neural collapse with unconstrained featurescited by 140
  11. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  12. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  13. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  14. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  15. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  16. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  17. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  18. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  19. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  20. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  21. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  22. 2020Neural Collapse with Cross-Entropy Losscited by 52
  23. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  24. 2020Grafit: Learning fine-grained image representations with coarse labelscited by 43
  25. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  26. 2022A Law of Data Separation in Deep Learningcited by 36
  27. 2021Rethinking Supervised Pre-training for Better Downstream Transferringcited by 35
  28. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  29. 2022Perturbation Analysis of Neural Collapsecited by 34
  30. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  31. 2018How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networkscited by 28
  32. 2020Separation and Concentration in Deep Networkscited by 28
  33. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23
  34. 2019A Group-Theoretic Framework for Data Augmentationcited by 14
  35. 2022Discriminability-Transferability Trade-Off: An Information-Theoretic Perspectivecited by 13
  36. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  37. 2022Learning Towards the Largest Marginscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 2.

9 papers.

  1. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  2. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  3. 2025MaxSup: Overcoming Representation Collapse in Label Smoothingcited by 4
  4. 2025Balancing Two Classifiers via A Simplex ETF Structure for Model Calibrationcited by 3
  5. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  6. 2025Exploring and Leveraging Class Vectors for Classifier Editingcited by 1
  7. 2026The Implicit Bias of Logit Regularizationcited by 1
  8. 2025Contrastive Self-Supervised Learning at the Edge: An Energy Perspectivecited by 0
  9. 2026Empirical Study of Observable Sets in Multiclass Quantum Classificationcited by 0