mlsanityData to 14 Oct 2026

Phase Diagram of Initial Condensation for Two-layer Neural Networks

arXiv 2303.06561 ↗First posted 12 Mar 2023cs.LG cond-mat.dis-nn math.OCcited by 16

Not on the map. Why?

Abstract (arXiv)

The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the earlier work by Luo et al.~citeluo2021phase, we present a phase diagram of initial condensation for two-layer neural networks. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations during the training process, and it is a feature in non-linear learning process that enables neural networks to possess better generalization abilities. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks and their dependence on the choice of hyperparameters related to initialization. Furthermore, we demonstrate in detail the underlying mechanisms by which small initialization leads to condensation at the initial training stage.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

21 of 32 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  7. 2015An Introduction to Convolutional Neural Networkscited by 514
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  11. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  12. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
9 more
  1. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  2. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  3. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  4. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  5. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  6. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  7. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
  8. 2020An analytic theory of shallow networks dynamics for hinge loss classificationcited by 14
  9. 2020Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 5; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  2. 2025An overview of condensation phenomenon in deep learningcited by 17
  3. 2023Loss Spike in Training Neural Networkscited by 16
  4. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  5. 2024Anchor function: a type of benchmark functions for studying language modelscited by 11
  6. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11
  7. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 10
  8. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 8
  9. 2024On the dynamics of three-layer neural networks: initial condensationcited by 3
  10. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 1
  11. 2025KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approachescited by 1
  12. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
4 more
  1. 2025Statistical Learning Guarantees for Group-Invariant Barron Functionscited by 1
  2. 2024Demystifying Lazy Training of Neural Networks from a Macroscopic Viewpointcited by 0
  3. 2026Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetrycited by 0
  4. 2026Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networkscited by 0