mlsanityData to 7 Oct 2026

Conservation Laws for Modern Neural Architectures

arXiv 2606.17816 ↗First posted 16 Jun 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

37 of 53 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
25 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2024Mixtral of Expertscited by 1,785
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  8. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2016Language Modeling with Gated Convolutional Networkscited by 688
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  13. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  14. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  15. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  16. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  17. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  18. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  19. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  20. 2023Transformers learn through gradual rank increasecited by 52
  21. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  22. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  23. 2019Equi-normalization of Neural Networkscited by 16
  24. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  25. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
  2. 2026The Forward-Backward Disconnect: State Dynamics, Credit Assignment, and Biological Grounding in Neural Computation ↗
  3. 2026Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
  4. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models
  5. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the story
  6. 2024Towards a theory of learning dynamics in deep state space models
  7. 2026Prediction Is Not Physics: Learning and Evaluating Conserved Quantities in Neural Simulators
  8. 2026Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective