# Regularization Implies balancedness in the deep linear network

arXiv: https://arxiv.org/abs/2511.01137 · HTML: https://mlsanity.com/paper/2511.01137/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Builds on (most cited first, 20 of 20)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics](https://mlsanity.com/paper/2012.04728/) (2020)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows](https://mlsanity.com/paper/2307.00144/) (2023)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Flat minima generalize for low-rank matrix recovery](https://mlsanity.com/paper/2203.03756/) (2022)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows](https://mlsanity.com/paper/2405.12888/) (2024)
- [The geometry of the deep linear network](https://mlsanity.com/paper/2411.09004/) (2024)
- [An entropy formula for the Deep Linear Network](https://mlsanity.com/paper/2509.09088/) (2025)
- [On the Implicit Bias of Gradient Descent for Temporal Extrapolation](https://mlsanity.com/paper/2202.04302/) (2022)
- [Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States](https://mlsanity.com/paper/2402.07875/) (2024)
- [Geometry of fibers of the multiplication map of deep linear neural networks](https://mlsanity.com/paper/2411.19920/) (2024)
- [The Geometry of the Set of Equivalent Linear Neural Networks](https://mlsanity.com/paper/2404.14855/) (2024)
- [On Functional Dimension and Persistent Pseudodimension](https://mlsanity.com/paper/2410.17191/) (2024)

## Built on by (most cited first, 4 of 4)

- [Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks](https://mlsanity.com/paper/2608.06597/) (2026)
- [Benign Loss Landscapes Can Coexist with Worst-Case Hardness](https://mlsanity.com/paper/2609.13057/) (2026)
- [Special Lagrangian cones in Deep Learning](https://mlsanity.com/paper/2609.20159/) (2026)
- [On Parameter Symmetries and Conservation Laws in Gradient Flow](https://mlsanity.com/paper/2609.34549/) (2026)

## Abstract (arXiv)

We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the L2 regularizer is minimized on the balanced manifold. We introduce related balancing flows using the Riemannian geometry of fibers. The balancing flow defined by the L2 regularizer is shown to converge to the balanced manifold at a uniform exponential rate. The balancing flow defined by the squared moment map is computed explicitly and shown to converge globally. This framework allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. It also provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of fast-slow systems, model reduction and Bayesian principles.