# Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows

arXiv: https://arxiv.org/abs/2405.12888 · HTML: https://mlsanity.com/paper/2405.12888/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 17 of 17)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Optimal transport mapping via input convex neural networks](https://mlsanity.com/paper/1908.10962/) (2019)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [A Variational Perspective on Accelerated Methods in Optimization](https://mlsanity.com/paper/1603.04245/) (2016)
- [Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics](https://mlsanity.com/paper/2012.04728/) (2020)
- [Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers](https://mlsanity.com/paper/1910.05505/) (2019)
- [The Information Geometry of Mirror Descent](https://mlsanity.com/paper/1310.7780/) (2013)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows](https://mlsanity.com/paper/2307.00144/) (2023)
- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [Noether: The More Things Change, the More Stay the Same](https://mlsanity.com/paper/2104.05508/) (2021)

## Built on by (most cited first, 11 of 11)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Transformative or Conservative? Conservation laws for ResNets and Transformers](https://mlsanity.com/paper/2506.06194/) (2025)
- [Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch](https://mlsanity.com/paper/2504.12801/) (2025)
- [Intrinsic training dynamics of deep neural networks](https://mlsanity.com/paper/2508.07370/) (2025)
- [Regularization Implies balancedness in the deep linear network](https://mlsanity.com/paper/2511.01137/) (2025)
- [Never Saddle for Reparameterized Steepest Descent as Mirror Flow](https://mlsanity.com/paper/2603.02064/) (2026)
- [Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning](https://mlsanity.com/paper/2605.19458/) (2026)
- [Optimization Insights into Deep Diagonal Linear Networks](https://mlsanity.com/paper/2412.16765/) (2024)
- [Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries](https://mlsanity.com/paper/2607.23967/) (2026)
- [Conservation Laws for Modern Neural Architectures](https://mlsanity.com/paper/2606.17816/) (2026)
- [On Parameter Symmetries and Conservation Laws in Gradient Flow](https://mlsanity.com/paper/2609.34549/) (2026)

## Abstract (arXiv)

Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles for non-Euclidean geometries and momentum-based dynamics remain largely unknown. In this paper, we characterize "all" conservation laws in this general setting. In stark contrast to the case of gradient flows, we prove that the conservation laws for momentum-based dynamics exhibit temporal dependence. Additionally, we often observe a "conservation loss" when transitioning from gradient flow to momentum dynamics. Specifically, for linear networks, our framework allows us to identify all momentum conservation laws, which are less numerous than in the gradient flow case except in sufficiently over-parameterized regimes. With ReLU networks, no conservation law remains. This phenomenon also manifests in non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF): all conservation laws can be determined in the gradient flow context, yet none persists in the momentum case.