mlsanityData to 14 Oct 2026

The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

arXiv 2405.15706 ↗First posted 24 May 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 49 of its 67 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

49 of 67 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2016Matching Networks for One Shot Learningcited by 2,255
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2014How transferable are features in deep neural networks?cited by 1,225
  11. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  12. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
37 more
  1. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  2. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  4. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  5. 2014The Loss Surfaces of Multilayer Networkscited by 280
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  8. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  9. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  10. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  11. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  12. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  13. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  14. 2020Implicit Gradient Regularizationcited by 146
  15. 2019Robust Learning with Jacobian Regularizationcited by 129
  16. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  17. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  18. 2016Robust Large Margin Deep Neural Networkscited by 118
  19. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  20. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  21. 2017Implicit Regularization in Deep Learningcited by 90
  22. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  23. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  24. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  25. 2020A case for new neural network smoothness constraintscited by 43
  26. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  27. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  28. 2023On the Implicit Bias of Adamcited by 27
  29. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  30. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  31. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  32. 2021A Concise Review of Transfer Learningcited by 13
  33. 2021The Geometric Occam's Razor Implicit in Deep Learningcited by 7
  34. 2023Towards Demystifying the Generalization Behaviors When Neural Collapse Emergescited by 7
  35. 2019Statistical Estimation of the Poincar'e constant and Application to Sampling Multimodal Distributionscited by 5
  36. 2021Efficient approximation of Jacobian matrices involving a non-uniform fast Fourier transform (NUFFT)cited by 5
  37. 2021Functional Space Analysis of Local GAN Convergencecited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  2. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  3. 2024A Bayesian Model Selection Criterion for Selecting Pretraining Checkpointscited by 5
  4. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3