mlsanityData to 7 Oct 2026

Regularization Implies balancedness in the deep linear network

arXiv 2511.01137 ↗First posted 3 Nov 2025cs.LG math.AG math.DScited by 4

Not on the map. Why?

Abstract (arXiv)

We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the L2 regularizer is minimized on the balanced manifold. We introduce related balancing flows using the Riemannian geometry of fibers. The balancing flow defined by the L2 regularizer is shown to converge to the balanced manifold at a uniform exponential rate. The balancing flow defined by the squared moment map is computed explicitly and shown to converge globally. This framework allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. It also provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of fast-slow systems, model reduction and Bayesian principles.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 47 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

20 of 47 references.

References · 20

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  3. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  6. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  7. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  8. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  9. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  10. 2025Symmetry in Neural Network Parameter Spacescited by 31
  11. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  12. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
8 more
  1. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  2. 2024The geometry of the deep linear networkcited by 10
  3. 2025An entropy formula for the Deep Linear Networkcited by 8
  4. 2022On the Implicit Bias of Gradient Descent for Temporal Extrapolationcited by 7
  5. 2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescited by 4
  6. 2024Geometry of fibers of the multiplication map of deep linear neural networkscited by 4
  7. 2024The Geometry of the Set of Equivalent Linear Neural Networkscited by 2
  8. 2024On Functional Dimension and Persistent Pseudodimensioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0
  2. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0
  3. 2026Special Lagrangian cones in Deep Learningcited by 0
  4. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0