mlsanityData to 14 Oct 2026

Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway

arXiv 2606.05219 ↗First posted 29 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent analyses of multi-pathway Deep Linear Networks use Gradient Flow to predict a "winner-takes-all" specialization in which path symmetry breaks and each feature concentrates in a single pathway. In this work, we show that discrete Gradient Descent (GD) with a large step size tells a different story. We prove that single-path solutions are sharp minima, whereas distributing signals across pathways reduces sharpness by a factor that decreases with both the number of pathways and depth. Consequently, while early training reproduces the depth-driven symmetry breaking predicted by GF, oscillations at the Edge of Stability subsequently override this tendency and drive the network into a re-balancing phase, where signals redistribute across pathways. Together, these results clarify how depth shapes pathway competition and explain why large-step GD favors shared representations rather than persistent single-pathway dominance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

21 of 33 references.

References · 21

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  5. 2017Exploring Generalization in Deep Learningcited by 600
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  8. 2017Implicit Regularization in Matrix Factorizationcited by 322
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  11. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
9 more
  1. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  2. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  3. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  4. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  5. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  6. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  7. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  8. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  9. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 9

Built on by

Not cited yet.