mlsanityData to 7 Oct 2026

An overview of condensation phenomenon in deep learning

arXiv 2504.09484 ↗First posted 13 Apr 2025cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs. Empirical observations suggest that the number of condensed clusters of neurons in the same layer typically increases monotonically as training progresses. Neural networks with small weight initializations or Dropout optimization can facilitate this condensation process. We also examine the underlying mechanisms of condensation from the perspectives of training dynamics and the structure of the loss landscape. The condensation phenomenon offers valuable insights into the generalization abilities of neural networks and correlates to stronger reasoning abilities in transformer-based language models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

43 of 59 references.

References · 43

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018On the Spectral Bias of Neural Networkscited by 1,321
  8. 2017A Closer Look at Memorization in Deep Networkscited by 936
  9. 2018On Lazy Training in Differentiable Programmingcited by 671
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  11. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  12. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
31 more
  1. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  2. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  3. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  4. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  5. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
  6. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  7. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  8. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  9. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  10. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  11. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  12. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 29
  13. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  14. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  15. 2022Implicit regularization of dropoutcited by 24
  16. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  17. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  18. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  19. 2024Early alignment in two-layer networks training is a two-edged swordcited by 17
  20. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  21. 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 15
  22. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  23. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  24. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  25. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  26. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 10
  27. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 7
  28. 2024Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializationscited by 6
  29. 2023Optimistic Estimate Uncovers the Potential of Nonlinear Modelscited by 5
  30. 2024On the dynamics of three-layer neural networks: initial condensationcited by 3
  31. 2024Efficient and Flexible Method for Reducing Moderate-size Deep Neural Networks with Condensationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  2. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 9
  3. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  4. 2025Probability Signature: Bridging Data Semantics and Embedding Structure in Language Modelscited by 2
  5. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  6. 2026Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorizecited by 1
  7. 2025KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approachescited by 0
  8. 2025KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networkscited by 0
  9. 2025Limit Analysis for Symbolic Multi-step Reasoning Tasks with Information Propagation Rules Based on Transformerscited by 0
  10. 2025Noise-robust Contrastive Learning for Critical Transition Detection in Dynamical Systemscited by 0
  11. 2026Sparse RBF Networks for PDEs and nonlocal equations: function space theory, operator calculus, and training algorithmscited by 0
  12. 2026Small Initialization Matters for Large Language Modelscited by 0
1 more
  1. 2026Starter-Iterator Neural Operator: A Unified Architecture for High-Fidelity Forward and Inverse PDE Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models
  2. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective
  3. 2024Learning Dynamics of LLM Finetuning
  4. 2025Characterising the Inductive Biases of Neural Networks on Boolean Data
  5. 2026Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
  6. 2026A Theory of Generalization in Deep Learning
  7. 2023Stabilizing RNN Gradients through Pre-training
  8. 2026Memorisation, convergence and generalisation in generative models