mlsanityData to 7 Oct 2026

Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

arXiv 2309.04644 ↗First posted 9 Sep 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural Collapse (NC) is a geometric structure recently observed at the terminal phase of training deep neural networks, which states that last-layer feature vectors for the same class would "collapse" to a single point, while features of different classes become equally separated. We demonstrate that batch normalization (BN) and weight decay (WD) critically influence the emergence of NC. In the near-optimal loss regime, we establish an asymptotic lower bound on the emergence of NC that depends only on the WD value, training loss, and the presence of last-layer BN. Our experiments substantiate theoretical insights by showing that models demonstrate a stronger presence of NC with BN, appropriate WD values, lower loss, and lower last-layer feature norm. Our findings offer a novel perspective in studying the role of BN and WD in shaping neural network features.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

29 of 58 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  6. 2018How Does Batch Normalization Help Optimization?cited by 355
  7. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  8. 2018Understanding Batch Normalizationcited by 157
  9. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  10. 2020Neural collapse with unconstrained featurescited by 140
  11. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  12. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
17 more
  1. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  2. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  3. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  4. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  5. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  6. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  7. 2019A Mean Field Theory of Batch Normalizationcited by 58
  8. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  9. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  10. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  11. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  12. 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
  13. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  14. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  15. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  16. 2022Batch Normalization Explainedcited by 8
  17. 2015On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Unitscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 3.

9 papers.

  1. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  2. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  3. 2025Cautious Weight Decaycited by 12
  4. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  5. 2024The Fair Language Model Paradoxcited by 3
  6. 2026Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Thresholdcited by 2
  7. 2026Optimizer choice matters for the emergence of Neural Collapsecited by 1
  8. 2026Deep Neural Regression Collapsecited by 1
  9. 2025TRUST: Test-time Resource Utilization for Superior Trustworthinesscited by 0