# Towards Understanding the Condensation of Neural Networks at Initial Training

arXiv: https://arxiv.org/abs/2105.11686 · HTML: https://mlsanity.com/paper/2105.11686/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 25 May 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel](https://mlsanity.com/paper/2010.15110/) (2020)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Mean Field Analysis of Neural Networks: A Central Limit Theorem](https://mlsanity.com/paper/1808.09372/) (2018)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Gradient Descent Quantizes ReLU Network Features](https://mlsanity.com/paper/1803.08367/) (2018)
- [The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks](https://mlsanity.com/paper/2006.14599/) (2020)
- [A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics](https://mlsanity.com/paper/1904.04326/) (2019)
- [A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems](https://mlsanity.com/paper/1909.11759/) (2019)
- [A type of generalization error induced by initialization in deep neural networks](https://mlsanity.com/paper/1905.07777/) (2019)
- [Deep frequency principle towards understanding why deeper learning is faster](https://mlsanity.com/paper/2007.14313/) (2020)
- [Semi-flat minima and saddle points by embedding neural networks to overparameterization](https://mlsanity.com/paper/1906.04868/) (2019)
- [Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness](https://mlsanity.com/paper/1905.11427/) (2019)
- [An analytic theory of shallow networks dynamics for hinge loss classification](https://mlsanity.com/paper/2006.11209/) (2020)
- [The Slow Deterioration of the Generalization Error of the Random Feature Model](https://mlsanity.com/paper/2008.05621/) (2020)

## Built on by (most cited first, 30 of 32)

- [Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective](https://mlsanity.com/paper/2502.00604/) (2025)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers](https://mlsanity.com/paper/2501.08537/) (2025)
- [Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width](https://mlsanity.com/paper/2205.12101/) (2022)
- [Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing](https://mlsanity.com/paper/2405.05409/) (2024)
- [An overview of condensation phenomenon in deep learning](https://mlsanity.com/paper/2504.09484/) (2025)
- [Loss Spike in Training Neural Networks](https://mlsanity.com/paper/2305.12133/) (2023)
- [Anchor function: a type of benchmark functions for studying language models](https://mlsanity.com/paper/2401.08309/) (2024)
- [Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data](https://mlsanity.com/paper/2405.17299/) (2024)
- [From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics](https://mlsanity.com/paper/2510.06954/) (2025)
- [Understanding the Initial Condensation of Convolutional Neural Networks](https://mlsanity.com/paper/2305.09947/) (2023)
- [Linear Stability Hypothesis and Rank Stratification for Nonlinear Models](https://mlsanity.com/paper/2211.11623/) (2022)
- [Optimistic Estimate Uncovers the Potential of Nonlinear Models](https://mlsanity.com/paper/2307.08921/) (2023)
- [On the dynamics of three-layer neural networks: initial condensation](https://mlsanity.com/paper/2402.15958/) (2024)
- [Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion](https://mlsanity.com/paper/2405.13721/) (2024)
- [Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?](https://mlsanity.com/paper/2409.16697/) (2024)
- [On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages](https://mlsanity.com/paper/2410.20119/) (2024)
- [KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches](https://mlsanity.com/paper/2505.14777/) (2025)
- [Scalable Complexity Control Facilitates Reasoning Ability of LLMs](https://mlsanity.com/paper/2505.23013/) (2025)
- [Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize](https://mlsanity.com/paper/2605.04396/) (2026)
- [Loss Jump During Loss Switch in Solving PDEs with Neural Networks](https://mlsanity.com/paper/2405.03095/) (2024)
- [Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Target](https://mlsanity.com/paper/2405.13787/) (2024)
- [Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterization](https://mlsanity.com/paper/2406.18035/) (2024)
- [pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning](https://mlsanity.com/paper/2409.05701/) (2024)
- [Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks](https://mlsanity.com/paper/2504.08628/) (2025)
- [KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks](https://mlsanity.com/paper/2505.17919/) (2025)
- [DeePoly: A High-Order Accuracy Scientific Machine Learning Framework for Function Approximation and Solving PDEs](https://mlsanity.com/paper/2506.04613/) (2025)
- [Machine Learning and Control: Foundations, Advances, and Perspectives](https://mlsanity.com/paper/2510.03303/) (2025)
- [The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training](https://mlsanity.com/paper/2605.26489/) (2026)
- [Small Initialization Matters for Large Language Models](https://mlsanity.com/paper/2606.17945/) (2026)

## Abstract (arXiv)

Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations. The condensation dynamics implies that the training implicitly regularizes a NN towards one with a much smaller effective size. In this work, we illustrate the formation of the condensation in multi-layer fully connected NNs and show that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function, where "multiplicity" indicates the multiple roots of activation function at origin. Our theoretical analysis confirms experiments for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity. This work makes a step towards understanding how small initialization leads NNs to condensation at the initial training stage.