# An Equivariance Toolbox for Learning Dynamics

arXiv: https://arxiv.org/abs/2512.21447 · HTML: https://mlsanity.com/paper/2512.21447/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 24 Dec 2025.

## Builds on (most cited first, 30 of 30)

- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics](https://mlsanity.com/paper/2012.04728/) (2020)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape](https://mlsanity.com/paper/1907.02911/) (2019)
- [Gradient Descent Quantizes ReLU Network Features](https://mlsanity.com/paper/1803.08367/) (2018)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks](https://mlsanity.com/paper/2409.14623/) (2024)
- [Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks](https://mlsanity.com/paper/2010.04261/) (2020)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization](https://mlsanity.com/paper/2307.12851/) (2023)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [Noether: The More Things Change, the More Stay the Same](https://mlsanity.com/paper/2104.05508/) (2021)
- [Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width](https://mlsanity.com/paper/2302.12250/) (2023)
- [Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent](https://mlsanity.com/paper/2402.07193/) (2024)
- [Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry](https://mlsanity.com/paper/2008.01805/) (2020)
- [Deep linear networks for regression are implicitly regularized towards flat minima](https://mlsanity.com/paper/2405.13456/) (2024)
- [Improving Optimization for Models With Continuous Symmetry Breaking](https://mlsanity.com/paper/1803.03234/) (2018)
- [Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More](https://mlsanity.com/paper/2506.06940/) (2025)

## Abstract (arXiv)

Many theoretical results in deep learning can be traced to symmetry or equivariance of neural networks under parameter transformations. However, existing analyses are typically problem-specific and focus on first-order consequences such as conservation laws, while the implications for second-order structure remain less understood. We develop a general equivariance toolbox that yields coupled first- and second-order constraints on learning dynamics. The framework extends classical Noether-type analyses in three directions: from gradient constraints to Hessian constraints, from symmetry to general equivariance, and from continuous to discrete transformations. At the first order, our framework unifies conservation laws and implicit-bias relations as special cases of a single identity. At the second order, it provides structural predictions about curvature: which directions are flat or sharp, how the gradient aligns with Hessian eigenspaces, and how the loss landscape geometry reflects the underlying transformation structure. We illustrate the framework through several applications, recovering known results while also deriving new characterizations that connect transformation structure to modern empirical observations about optimization geometry.