mlsanityData to 7 Oct 2026

Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows

arXiv 2307.00144 ↗First posted 30 Jun 2023cs.LG math.OCcited by 44

Not on the map. Why?

Abstract (arXiv)

Understanding the geometric properties of gradient descent dynamics is a key ingredient in deciphering the recent success of very large machine learning models. A striking observation is that trained over-parameterized models retain some properties of the optimization initialization. This "implicit bias" is believed to be responsible for some favorable properties of the trained models and could explain their good generalization properties. The purpose of this article is threefold. First, we rigorously expose the definition and basic properties of "conservation laws", that define quantities conserved during gradient flows of a given model (e.g. of a ReLU network with a given architecture) with any training data and any loss. Then we explain how to find the maximal number of independent conservation laws by performing finite-dimensional algebraic manipulations on the Lie algebra generated by the Jacobian of the model. Finally, we provide algorithms to: a) compute a family of polynomial laws; b) compute the maximal number of (not necessarily polynomial) independent conservation laws. We provide showcase examples that we fully work out theoretically. Besides, applying the two algorithms confirms for a number of ReLU network architectures that all known laws are recovered by the algorithm, and that there are no other independent laws. Such computational tools pave the way to understanding desirable properties of optimization initialization in large machine learning models.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

21 of 36 references.

References · 21

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  12. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
9 more
  1. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  2. 2017Implicit Regularization in Deep Learningcited by 90
  3. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  4. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  5. 2020Gradient descent follows the regularization path for general lossescited by 57
  6. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  7. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  8. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  9. 2021Noether: The More Things Change, the More Stay the Samecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 13; 2026: 22.

44 papers.

Sort:

  1. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  2. 2025Symmetry in Neural Network Parameter Spacescited by 31
  3. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  5. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  6. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  7. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  8. 2023A path-norm toolkit for modern networks: consequences, promises and challengescited by 10
  9. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  10. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  11. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  12. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7
32 more
  1. 2025Sign-In to the Lottery: Reparameterizing Sparse Training From Scratchcited by 6
  2. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  3. 2026Path-conditioned training: a principled way to rescale ReLU neural networkscited by 6
  4. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  5. 2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescited by 4
  6. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  7. 2024A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizationscited by 4
  8. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  9. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  10. 2025Intrinsic training dynamics of deep neural networkscited by 3
  11. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  12. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  13. 2025Convergence of Shallow ReLU Networks on Weakly Interacting Datacited by 2
  14. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  15. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  16. 2026The Symmetries of Three-Layer ReLU Networkscited by 2
  17. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
  18. 2026Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetriescited by 2
  19. 2025Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?cited by 1
  20. 2026Conservation Laws from Data Symmetry in Neural Networkscited by 1
  21. 2026Conservation Laws for Modern Neural Architecturescited by 1
  22. 2026Hidden Gauge Controls Feature Specialization in ReLU Networkscited by 1
  23. 2026Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimizationcited by 0
  24. 2026Three-Phase Transformercited by 0
  25. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  26. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 0
  27. 2026Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamicscited by 0
  28. 2026Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networkscited by 0
  29. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0
  30. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0
  31. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0
  32. 2026Universal interpolation for deep residual self-attention networkscited by 0