# Phase Collapse in Neural Networks

arXiv: https://arxiv.org/abs/2110.05283 · HTML: https://mlsanity.com/paper/2110.05283/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Robust and interpretable blind image denoising via bias-free convolutional neural networks](https://mlsanity.com/paper/1906.05478/) (2019)
- [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://mlsanity.com/paper/1603.05201/) (2016)
- [Kymatio: Scattering Transforms in Python](https://mlsanity.com/paper/1812.11214/) (2018)
- [Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?](https://mlsanity.com/paper/1504.08291/) (2015)
- [Separation and Concentration in Deep Networks](https://mlsanity.com/paper/2012.10424/) (2020)
- [DCFNet: Deep Neural Network with Decomposed Convolutional Filters](https://mlsanity.com/paper/1802.04145/) (2018)
- [On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks](https://mlsanity.com/paper/1806.00701/) (2018)
- [Deep Network Classification by Scattering and Homotopy Dictionary Learning](https://mlsanity.com/paper/1910.03561/) (2019)
- [Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning](https://mlsanity.com/paper/1708.08705/) (2017)
- [A mathematical motivation for complex-valued convolutional networks](https://mlsanity.com/paper/1503.03438/) (2015)
- [Deep Dictionary Learning: A PARametric NETwork Approach](https://mlsanity.com/paper/1803.04022/) (2018)
- [A Learnable ScatterNet: Locally Invariant Convolutional Layers](https://mlsanity.com/paper/1903.03137/) (2019)

## Built on by (most cited first, 5 of 5)

- [White-Box Transformers via Sparse Rate Reduction](https://mlsanity.com/paper/2306.01129/) (2023)
- [Emergence of Segmentation with Minimalistic White-Box Transformers](https://mlsanity.com/paper/2308.16271/) (2023)
- [Scaling Spherical CNNs](https://mlsanity.com/paper/2306.05420/) (2023)
- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures](https://mlsanity.com/paper/2610.04413/) (2026)

## Abstract (arXiv)

Deep convolutional classifiers linearly separate image classes and improve accuracy as depth increases. They progressively reduce the spatial dimension whereas the number of channels grows with depth. Spatial variability is therefore transformed into variability along channels. A fundamental challenge is to understand the role of non-linearities together with convolutional filters in this transformation. ReLUs with biases are often interpreted as thresholding operators that improve discrimination through sparsity. This paper demonstrates that it is a different mechanism called phase collapse which eliminates spatial variability while linearly separating classes. We show that collapsing the phases of complex wavelet coefficients is sufficient to reach the classification accuracy of ResNets of similar depths. However, replacing the phase collapses with thresholding operators that enforce sparsity considerably degrades the performance. We explain these numerical results by showing that the iteration of phase collapses progressively improves separation of classes, as opposed to thresholding non-linearities.