Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks
Not on the map. Why?
Abstract (arXiv)
Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 3 Oct 2026.
Builds on
29 of 32 references.
References · 29
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
- 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
- 2019How do infinite width bounded norm networks look in function space?cited by 106
- 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
- 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
- 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
17 more
- 2022Overview frequency principle/spectral bias in deep learningcited by 69
- 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
- 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
- 2020On the training dynamics of deep networks with L2 regularizationcited by 39
- 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
- 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
- 2022Implicit regularization of dropoutcited by 25
- 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
- 2024Early alignment in two-layer networks training is a two-edged swordcited by 18
- 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
- 2025An overview of condensation phenomenon in deep learningcited by 17
- 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 16
- 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 16
- 2024Anchor function: a type of benchmark functions for studying language modelscited by 11
- 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11
- 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 10
- 2025Low-rank bias, weight decay, and model merging in neural networkscited by 5
Built on by
Not cited yet.