mlsanityData to 14 Oct 2026

On Parameter Symmetries and Conservation Laws in Gradient Flow

arXiv 2609.34549 ↗First posted 28 Sep 2026cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 64 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

44 of 64 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2014Going Deeper with Convolutionscited by 4,241
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  11. 2018How Does Batch Normalization Help Optimization?cited by 357
  12. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
32 more
  1. 2015Bilinear CNNs for Fine-grained Visual Recognitioncited by 228
  2. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  3. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  4. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  5. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 110
  6. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  7. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  8. 2019Optimization for deep learning: theory and algorithmscited by 70
  9. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 66
  10. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  11. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
  12. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  13. 2023Hidden symmetries of ReLU networkscited by 39
  14. 2025Symmetry in Neural Network Parameter Spacescited by 36
  15. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  16. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  17. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  18. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  19. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  20. 2024Equivariant Neural Functional Networks for Transformerscited by 22
  21. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  22. 2020Affine symmetries and neural network identifiabilitycited by 18
  23. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  24. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  25. 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
  26. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7
  27. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  28. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  29. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  30. 2026The Symmetries of Three-Layer ReLU Networkscited by 3
  31. 2026Conservation Laws from Data Symmetry in Neural Networkscited by 1
  32. 2026Conservation Laws for Modern Neural Architecturescited by 1

Built on by

Not cited yet.