mlsanityData to 7 Oct 2026

Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks

arXiv 2105.02716 ↗First posted 6 May 2021cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 47

Not on the map. Why?

Abstract (arXiv)

In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture regularities in the world, but the role of symmetry breaking is not well understood. Here, we develop a theoretical framework to study the "geometry of learning dynamics" in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify "kinetic symmetry breaking" (KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether's theorem known in physics to take into account KSB and derive the resulting motion of the Noether charge: "Noether's Learning Dynamics" (NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces a mechanism of "implicit adaptive optimization", establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 52 of its 79 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

52 of 79 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2018Group Normalizationcited by 1,514
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
38 more
  1. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  2. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  3. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  5. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  6. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  7. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  8. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  9. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  10. 2018Understanding Batch Normalizationcited by 157
  11. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  12. 2020Implicit Gradient Regularizationcited by 145
  13. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  14. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  15. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  16. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  17. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  18. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  19. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  20. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  21. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  22. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  23. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  24. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  25. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  26. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  27. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 45
  28. 2017Riemannian approach to batch normalizationcited by 35
  29. 2019Continuous Time Analysis of Momentum Methodscited by 32
  30. 2020The Two Regimes of Deep Network Trainingcited by 30
  31. 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
  32. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  33. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  34. 2015Understanding symmetries in deep networkscited by 17
  35. 2018Accelerating Natural Gradient with Higher-Order Invariancecited by 17
  36. 2021How to decay your learning ratecited by 11
  37. 2018Improving Optimization for Models With Continuous Symmetry Breakingcited by 8
  38. 2015On Accelerated Methods in Optimizationcited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 2; 2024: 10; 2025: 12; 2026: 16.

47 papers.

Sort:

By area:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2025Transformers without Normalizationcited by 69
  3. 2022Mechanistic Mode Connectivitycited by 50
  4. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  5. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  6. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  7. 2025Symmetry in Neural Network Parameter Spacescited by 31
  8. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  9. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  10. 2025Stronger Normalization-Free Transformerscited by 13
  11. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  12. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
35 more
  1. 2024Coding schemes in neural networks learning classification taskscited by 11
  2. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  3. 2024On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theorycited by 10
  4. 2024CoMMIT: Coordinated Multimodal Instruction Tuningcited by 8
  5. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  6. 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
  7. 2024Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODEcited by 6
  8. 2024Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equationscited by 6
  9. 2022Exact Phase Transitions in Deep Learningcited by 5
  10. 2025Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rulescited by 4
  11. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  12. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  13. 2026On the Spatiotemporal Dynamics of Generalization in Neural Networkscited by 3
  14. 2024Symmetry Breaking in Neural Network Optimization: Insights from Input Dimension Expansioncited by 2
  15. 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 2
  16. 2025Dynamical symmetries in the fluctuation-driven regime: an application of Noether's theorem to noisy dynamical systemscited by 2
  17. 2025Improving Learning to Optimize Using Parameter Symmetriescited by 2
  18. 2026Symmetry Breaking in Transformers for Efficient and Interpretable Trainingcited by 2
  19. 2026Toward Manifest Relationality in Transformers via Symmetry Reductioncited by 2
  20. 2026Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetriescited by 2
  21. 2026From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamicscited by 1
  22. 2026Minimum-Action Learning: Energy-Constrained Symbolic Model Selection for Physical Law Identification from Noisy Datacited by 1
  23. 2026Learning reveals invisible structure in low-rank RNNscited by 1
  24. 2026Conservation Laws from Data Symmetry in Neural Networkscited by 1
  25. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 1
  26. 2022Registering Image Volumes using 3D SIFT and Discrete SP-Symmetrycited by 0
  27. 2022U(1) Symmetry-breaking Observed in Generic CNN Bottleneck Layerscited by 0
  28. 2025Rolling Ball Optimizer: Learning by ironing out loss landscape wrinklescited by 0
  29. 2025Riemannian Time Warping: Multiple Sequence Alignment in Curved Spacescited by 0
  30. 2025A Variational Manifold Embedding Framework for Nonlinear Dimensionality Reductioncited by 0
  31. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 0
  32. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  33. 2026Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networkscited by 0
  34. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0
  35. 2026Why Adaptive Optimizers Underestimate Rare Tokenscited by 0