mlsanityData to 14 Oct 2026

Towards Understanding the Condensation of Neural Networks at Initial Training

arXiv 2105.11686 ↗First posted 25 May 2021cs.LG cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations. The condensation dynamics implies that the training implicitly regularizes a NN towards one with a much smaller effective size. In this work, we illustrate the formation of the condensation in multi-layer fully connected NNs and show that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function, where "multiplicity" indicates the multiple roots of activation function at origin. Our theoretical analysis confirms experiments for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity. This work makes a step towards understanding how small initialization leads NNs to condensation at the initial training stage.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

27 of 46 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  4. 2018On the Spectral Bias of Neural Networkscited by 1,341
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2017A Closer Look at Memorization in Deep Networkscited by 942
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  10. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  11. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  12. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
15 more
  1. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  2. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  3. 2018Training behavior of deep neural network in frequency domaincited by 174
  4. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  5. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 79
  6. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  7. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  8. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  9. 2019A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problemscited by 35
  10. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  11. 2020Deep frequency principle towards understanding why deeper learning is fastercited by 31
  12. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 30
  13. 2019Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothnesscited by 22
  14. 2020An analytic theory of shallow networks dynamics for hinge loss classificationcited by 14
  15. 2020The Slow Deterioration of the Generalization Error of the Random Feature Modelcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 11; 2025: 10; 2026: 5.

32 papers.

Sort:

By area:

  1. 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 56
  2. 2022Implicit regularization of dropoutcited by 25
  3. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  4. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  5. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
  6. 2025An overview of condensation phenomenon in deep learningcited by 17
  7. 2023Loss Spike in Training Neural Networkscited by 16
  8. 2024Anchor function: a type of benchmark functions for studying language modelscited by 11
  9. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  10. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 10
  11. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 8
  12. 2022Linear Stability Hypothesis and Rank Stratification for Nonlinear Modelscited by 7
20 more
  1. 2023Optimistic Estimate Uncovers the Potential of Nonlinear Modelscited by 5
  2. 2024On the dynamics of three-layer neural networks: initial condensationcited by 3
  3. 2024Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completioncited by 3
  4. 2024Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?cited by 1
  5. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 1
  6. 2025KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approachescited by 1
  7. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  8. 2026Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorizecited by 1
  9. 2024Loss Jump During Loss Switch in Solving PDEs with Neural Networkscited by 0
  10. 2024Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Targetcited by 0
  11. 2024Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterizationcited by 0
  12. 2024pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learningcited by 0
  13. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  14. 2025KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networkscited by 0
  15. 2025DeePoly: A High-Order Accuracy Scientific Machine Learning Framework for Function Approximation and Solving PDEscited by 0
  16. 2025Machine Learning and Control: Foundations, Advances, and Perspectivescited by 0
  17. 2026The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-trainingcited by 0
  18. 2026Small Initialization Matters for Large Language Modelscited by 0
  19. 2026Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetrycited by 0
  20. 2026Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networkscited by 0