mlsanityData to 14 Oct 2026

On the Role of Neural Collapse in Transfer Learning

arXiv 2112.15121 ↗First posted 30 Dec 2021cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that representations learned by a single classifier over many classes are competitive on few-shot learning problems with representations learned by special-purpose algorithms designed for such problems. In this paper we provide an explanation for this behavior based on the recently observed phenomenon that the features learned by overparameterized classification networks show an interesting clustering property, called neural collapse. We demonstrate both theoretically and empirically that neural collapse generalizes to new samples from the training classes, and -- more importantly -- to new classes as well, allowing foundation models to provide feature maps that work well in transfer learning and, specifically, in the few-shot setting.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

48 of 66 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2017Mask R-CNNcited by 7,277
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
36 more
  1. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  2. 2016Wide Residual Networkscited by 3,070
  3. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  4. 2016Matching Networks for One Shot Learningcited by 2,255
  5. 2014How transferable are features in deep neural networks?cited by 1,225
  6. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
  7. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  10. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  11. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  12. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  13. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  14. 2018Meta-learning with differentiable closed-form solverscited by 397
  15. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  16. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  17. 2019A Baseline for Few-Shot Image Classificationcited by 298
  18. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  19. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  20. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  21. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  22. 2017EMNIST: an extension of MNIST to handwritten letterscited by 213
  23. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 211
  24. 2015The Benefit of Multitask Representation Learningcited by 195
  25. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  26. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  27. 2020Neural collapse with unconstrained featurescited by 142
  28. 2017Few-Shot Image Recognition by Predicting Parameters from Activationscited by 122
  29. 2018Learning to Navigate for Fine-grained Classificationcited by 103
  30. 2020Interpretable and Accurate Fine-grained Recognition via Region Groupingcited by 75
  31. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  32. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  33. 2019Transductive Episodic-Wise Adaptive Metric for Few-Shot Learningcited by 67
  34. 2019Few-Shot Learning with Embedded Class Models and Shot-Free Meta Trainingcited by 57
  35. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  36. 2019Uniform concentration and symmetrization for weak interactionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 20; 2023: 19; 2024: 27; 2025: 16; 2026: 18.

100 papers.

Sort:

By area:

  1. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  2. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  3. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  4. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  5. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  6. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  7. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  8. 2023Demystifying Structural Disparity in Graph Neural Networks: Can One Size Fit All?cited by 41
  9. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  10. 2023Generalized Neural Collapse for a Large Number of Classescited by 39
  11. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  12. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
38 more
  1. 2022A Law of Data Separation in Deep Learningcited by 36
  2. 2022Perturbation Analysis of Neural Collapsecited by 35
  3. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  4. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  5. 2023Reverse Engineering Self-Supervised Learningcited by 33
  6. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  7. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  8. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  9. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  10. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  11. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  12. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
  13. 2024CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignmentcited by 17
  14. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  15. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  16. 2025Intermediate Layer Classifiers for OOD generalizationcited by 14
  17. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  18. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  19. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  20. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  21. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  22. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  23. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  24. 2024Coding schemes in neural networks learning classification taskscited by 11
  25. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  26. 2024EPA: Neural Collapse Inspired Robust Out-of-Distribution Detectorcited by 10
  27. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  28. 2024Formation of Representations in Neural Networkscited by 10
  29. 2023Quantifying the Variability Collapse of Neural Networkscited by 9
  30. 2023Differentially Private Video Activity Recognitioncited by 9
  31. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  32. 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 9
  33. 2024Words in Motion: Extracting Interpretable Control Vectors for Motion Transformerscited by 9
  34. 2024Control-oriented Clustering of Visual Latent Representationcited by 8
  35. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8
  36. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  37. 2022Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifierscited by 7
  38. 2023Towards Demystifying the Generalization Behaviors When Neural Collapse Emergescited by 7

50 most cited of 100 All 100 (JSON)