# Learning Dynamics of Deep Linear Networks Beyond the Edge of Stability

arXiv: https://arxiv.org/abs/2502.20531 · HTML: https://mlsanity.com/paper/2502.20531/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2025.

## Built on by (most cited first, 5 of 5)

- [Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence](https://mlsanity.com/paper/2605.31484/) (2026)
- [Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model](https://mlsanity.com/paper/2602.05065/) (2026)
- [Hard labels sampled from sparse targets mislead rotation invariant algorithms](https://mlsanity.com/paper/2603.20967/) (2026)
- [From the Loss Landscape to Diverse Feature Learning in Neural Networks](https://mlsanity.com/paper/2608.28948/) (2026)
- [The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow](https://mlsanity.com/paper/2609.01034/) (2026)

## Abstract (arXiv)

Deep neural networks trained using gradient descent with a fixed learning rate η often operate in the regime of "edge of stability" (EOS), where the largest eigenvalue of the Hessian equilibrates about the stability threshold 2/η. In this work, we present a fine-grained analysis of the learning dynamics of (deep) linear networks (DLNs) within the deep matrix factorization loss beyond EOS. For DLNs, loss oscillations beyond EOS follow a period-doubling route to chaos. We theoretically analyze the regime of the 2-period orbit and show that the loss oscillations occur within a small subspace, with the dimension of the subspace precisely characterized by the learning rate. The crux of our analysis lies in showing that the symmetry-induced conservation law for gradient flow, defined as the balancing gap among the singular values across layers, breaks at EOS and decays monotonically to zero. Overall, our results contribute to explaining two key phenomena in deep networks: (i) shallow models and simple tasks do not always exhibit EOS; and (ii) oscillations occur within top features. We present experiments to support our theory, along with examples demonstrating how these phenomena occur in nonlinear networks and how they differ from those which have benign landscape such as in DLNs.