# Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay

arXiv: https://arxiv.org/abs/2309.04644 · HTML: https://mlsanity.com/paper/2309.04644/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 9 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Understanding Batch Normalization](https://mlsanity.com/paper/1806.02375/) (2018)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features](https://mlsanity.com/paper/2203.01238/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [An Unconstrained Layer-Peeled Perspective on Neural Collapse](https://mlsanity.com/paper/2110.02796/) (2021)
- [Why Do Better Loss Functions Lead to Less Transferable Features?](https://mlsanity.com/paper/2010.16402/) (2020)
- [Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate](https://mlsanity.com/paper/2010.02916/) (2020)
- [A Mean Field Theory of Batch Normalization](https://mlsanity.com/paper/1902.08129/) (2019)
- [Towards Understanding Regularization in Batch Normalization](https://mlsanity.com/paper/1809.00846/) (2018)
- [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model](https://mlsanity.com/paper/2305.13165/) (2023)
- [Explicit regularization and implicit bias in deep network classifiers trained with the square loss](https://mlsanity.com/paper/2101.00072/) (2020)
- [Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization](https://mlsanity.com/paper/1805.10694/) (2018)
- [Nearest Class-Center Simplification through Intermediate Layers](https://mlsanity.com/paper/2201.08924/) (2022)
- [On the Implicit Bias Towards Minimal Depth of Deep Neural Networks](https://mlsanity.com/paper/2202.09028/) (2022)
- [Batch Normalization Explained](https://mlsanity.com/paper/2209.14778/) (2022)
- [On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Units](https://mlsanity.com/paper/1508.00330/) (2015)

## Built on by (most cited first, 9 of 9)

- [Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?](https://mlsanity.com/paper/2405.14468/) (2024)
- [Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse](https://mlsanity.com/paper/2410.04887/) (2024)
- [Cautious Weight Decay](https://mlsanity.com/paper/2510.12402/) (2025)
- [Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning](https://mlsanity.com/paper/2502.10691/) (2025)
- [The Fair Language Model Paradox](https://mlsanity.com/paper/2410.11985/) (2024)
- [Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold](https://mlsanity.com/paper/2604.00230/) (2026)
- [Optimizer choice matters for the emergence of Neural Collapse](https://mlsanity.com/paper/2602.16642/) (2026)
- [Deep Neural Regression Collapse](https://mlsanity.com/paper/2603.23805/) (2026)
- [TRUST: Test-time Resource Utilization for Superior Trustworthiness](https://mlsanity.com/paper/2506.06048/) (2025)

## Abstract (arXiv)

Neural Collapse (NC) is a geometric structure recently observed at the terminal phase of training deep neural networks, which states that last-layer feature vectors for the same class would "collapse" to a single point, while features of different classes become equally separated. We demonstrate that batch normalization (BN) and weight decay (WD) critically influence the emergence of NC. In the near-optimal loss regime, we establish an asymptotic lower bound on the emergence of NC that depends only on the WD value, training loss, and the presence of last-layer BN. Our experiments substantiate theoretical insights by showing that models demonstrate a stronger presence of NC with BN, appropriate WD values, lower loss, and lower last-layer feature norm. Our findings offer a novel perspective in studying the role of BN and WD in shaping neural network features.