mlsanityData to 13 Oct 2026

On the dynamics of three-layer neural networks: initial condensation

arXiv 2402.15958 ↗First posted 25 Feb 2024cs.LG math.DScited by 3

Not on the map. Why?

Abstract (arXiv)

Empirical and theoretical works show that the input weights of two-layer neural networks, when initialized with small values, converge towards isolated orientations. This phenomenon, referred to as condensation, indicates that the gradient descent methods tend to spontaneously reduce the complexity of neural networks during the training process. In this work, we elucidate the mechanisms behind the condensation phenomena occurring in the training of three-layer neural networks and distinguish it from the training of two-layer neural networks. Through rigorous theoretical analysis, we establish the blow-up property of effective dynamics and present a sufficient condition for the occurrence of condensation, findings that are substantiated by experimental results. Additionally, we explore the association between condensation and the low-rank bias observed in deep matrix factorization.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 18 of its 21 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

18 of 21 references.

References · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 465
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 425
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 404
  7. 2017Implicit Regularization in Matrix Factorizationcited by 321
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 305
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 282
  10. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  11. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  12. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
6 more
  1. 2018Gradient Descent Quantizes ReLU Network Featurescited by 59
  2. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  3. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
  4. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
  5. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  6. 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 16

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 55
  2. 2025An overview of condensation phenomenon in deep learningcited by 16
  3. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 0