mlsanityData to 14 Oct 2026

Neural (Tangent Kernel) Collapse

arXiv 2305.16427 ↗First posted 25 May 2023cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

This work bridges two important concepts: the Neural Tangent Kernel (NTK), which captures the evolution of deep neural networks (DNNs) during training, and the Neural Collapse (NC) phenomenon, which refers to the emergence of symmetry and structure in the last-layer features of well-trained classification DNNs. We adopt the natural assumption that the empirical NTK develops a block structure aligned with the class labels, i.e., samples within the same class have stronger correlations than samples from different classes. Under this assumption, we derive the dynamics of DNNs trained with mean squared (MSE) loss and break them into interpretable phases. Moreover, we identify an invariant that captures the essence of the dynamics, and use it to prove the emergence of NC in DNNs with block-structured NTK. We provide large-scale numerical experiments on three common DNN architectures and three benchmark datasets to support our theory.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

47 of 55 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2018On Lazy Training in Differentiable Programmingcited by 675
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  10. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 403
  11. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  12. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
35 more
  1. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  2. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  3. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  4. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  5. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  6. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  7. 2020Neural collapse with unconstrained featurescited by 142
  8. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  9. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  10. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  11. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  12. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  13. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  14. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  15. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  16. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  17. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  18. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  19. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  20. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  21. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  22. 2019The Local Elasticity of Neural Networkscited by 37
  23. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  24. 2022Perturbation Analysis of Neural Collapsecited by 35
  25. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  26. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  27. 2021SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEscited by 32
  28. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  29. 2021A Theory of Neural Tangent Kernel Alignment and Its Influence on Trainingcited by 25
  30. 2020Kernel-Based Smoothness Analysis of Residual Networkscited by 19
  31. 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
  32. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  33. 2021Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equationscited by 8
  34. 2019Neural Spectrum Alignment: Empirical Studycited by 6
  35. 2019Fix Your Features: Stationary and Maximally Discriminative Embeddings using Regular Polytope (Fixed Classifier) Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 5; 2026: 4.

18 papers.

Sort:

By area:

  1. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  2. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  3. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 22
  4. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 17
  5. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
  6. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 11
  7. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  8. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  9. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  10. 2025Theoretical Guarantees for Low-Rank Compression of Deep Neural Networkscited by 3
  11. 2025GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detectioncited by 2
  12. 2025Neural Tangent Knowledge Distillation for Optical Convolutional Networkscited by 2
6 more
  1. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  2. 2025OT Score: An OT based Confidence Score for Prototype-Assisted Source Free Unsupervised Domain Adaptationcited by 0
  3. 2026SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrizationcited by 0
  4. 2026Man, Machine, and Mathematicscited by 0
  5. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  6. 2026Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fieldscited by 0