mlsanityData to 7 Oct 2026

Progressive Feedforward Collapse of ResNet Training

arXiv 2405.00985 ↗First posted 2 May 2024cs.LG cs.AI math.OCcited by 12

Not on the map. Why?

Abstract (arXiv)

Neural collapse (NC) is a simple and symmetric phenomenon for deep neural networks (DNNs) at the terminal phase of training, where the last-layer features collapse to their class means and form a simplex equiangular tight frame aligning with the classifier vectors. However, the relationship of the last-layer features to the data and intermediate layers during training remains unexplored. To this end, we characterize the geometry of intermediate layers of ResNet and propose a novel conjecture, progressive feedforward collapse (PFC), claiming the degree of collapse increases during the forward propagation of DNNs. We derive a transparent model for the well-trained ResNet according to that ResNet with weight decay approximates the geodesic curve in Wasserstein space at the terminal phase. The metrics of PFC indeed monotonically decrease across depth on various datasets. We propose a new surrogate model, multilayer unconstrained feature model (MUFM), connecting intermediate layers by an optimal transport regularizer. The optimal solution of MUFM is inconsistent with NC but is more concentrated relative to the input data. Overall, this study extends NC to PFC to model the collapse phenomenon of intermediate layers and its dependence on the input data, shedding light on the theoretical understanding of ResNet in classification problems.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

46 of 59 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2016Deep Variational Information Bottleneckcited by 915
  6. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  10. 2017Stable Architectures for Deep Neural Networkscited by 340
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
34 more
  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  3. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  4. 2020Neural collapse with unconstrained featurescited by 140
  5. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  6. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  7. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  8. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  9. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  10. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  11. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  12. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  13. 2017Nonlinear Information Bottleneckcited by 71
  14. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  15. 2020Neural Collapse with Cross-Entropy Losscited by 52
  16. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  17. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  18. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  19. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  20. 2023Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Datacited by 40
  21. 2019Why ResNet Works? Residuals Generalizecited by 39
  22. 2022A Law of Data Separation in Deep Learningcited by 36
  23. 2022Perturbation Analysis of Neural Collapsecited by 34
  24. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  25. 2020Separation and Concentration in Deep Networkscited by 28
  26. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  27. 2022Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networkscited by 23
  28. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  29. 2019Towards Understanding the Importance of Shortcut Connections in Residual Networkscited by 18
  30. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  31. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  32. 2019Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Codingcited by 8
  33. 2021Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Spacecited by 8
  34. 2021Adversarial Information Bottleneckcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 5.

12 papers.

  1. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  2. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  3. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  4. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  5. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 9
  6. 2026Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholdcited by 2
  7. 2026Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmeticcited by 2
  8. 2024OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attackcited by 1
  9. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  10. 2025Are classical deep neural networks weakly adversarially robust?cited by 0
  11. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  12. 2026Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimizationcited by 0