Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
Not on the map. Why?
Abstract (arXiv)
We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which nested architectural symmetries force subnetworks to merge in discrete blocks rather than by single-edge attachment. These structural transitions register as variance spikes in a macroscopic order parameter echoing physical phase transitions. We further state sufficient conditions under which the trapping argument carries over to Adam and AdamW under heavy-tailed gradient noise and measure them on a trained Transformer.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 2 Sep 2026.
Builds on
32 of 61 references.
References · 32
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
20 more
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
- 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
- 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
- 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
- 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
- 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
- 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 62
- 2020Deep Learning is Singular, and That's Goodcited by 49
- 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
- 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
- 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
- 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
- 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
- 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 16
- 2020Anomalous diffusion dynamics of learning in deep neural networkscited by 9
- 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
- 2022Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamicscited by 3
- 2025Dropout Neural Network Training Viewed from a Percolation Perspectivecited by 2
Built on by
Not cited yet.