mlsanityData to 14 Oct 2026

Understanding the Initial Condensation of Convolutional Neural Networks

arXiv 2305.09947 ↗First posted 17 May 2023cs.LG cs.NEcited by 8

Not on the map. Why?

Abstract (arXiv)

Previous research has shown that fully-connected networks with small initialization and gradient-based training methods exhibit a phenomenon known as condensation during training. This phenomenon refers to the input weights of hidden neurons condensing into isolated orientations during training, revealing an implicit bias towards simple solutions in the parameter space. However, the impact of neural network structure on condensation has not been investigated yet. In this study, we focus on the investigation of convolutional neural networks (CNNs). Our experiments suggest that when subjected to small initialization and gradient-based training methods, kernel weights within the same CNN layer also cluster together during training, demonstrating a significant degree of condensation. Theoretically, we demonstrate that in a finite training period, kernels of a two-layer CNN with small initialization will converge to one or a few directions. This work represents a step towards a better understanding of the non-linear training behavior exhibited by neural networks with specialized structures.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

29 of 43 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  7. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  8. 2015Recent Advances in Convolutional Neural Networkscited by 360
  9. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  10. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  11. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  12. 2018Universality of Deep Convolutional Neural Networkscited by 176
17 more
  1. 2018Training behavior of deep neural network in frequency domaincited by 174
  2. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  3. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  4. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  5. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  6. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  7. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  8. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  9. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  10. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
  11. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 30
  12. 2022Implicit regularization of dropoutcited by 25
  13. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
  14. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  15. 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 16
  16. 2020An analytic theory of shallow networks dynamics for hinge loss classificationcited by 14
  17. 2022Linear Stability Hypothesis and Rank Stratification for Nonlinear Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 2.

8 papers.

  1. 2025An overview of condensation phenomenon in deep learningcited by 17
  2. 2024Anchor function: a type of benchmark functions for studying language modelscited by 11
  3. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 10
  4. 2024On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stagescited by 1
  5. 2026Focus and Dilution: The Multi-stage Learning Process of Attentioncited by 1
  6. 2026Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorizecited by 1
  7. 2023Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Frameworkcited by 0
  8. 2024Manifold Metric: A Loss Landscape Approach for Predicting Model Performancecited by 0