mlsanityData to 14 Oct 2026

Parameter Symmetry Potentially Unifies Deep Learning Theory

arXiv 2502.05300 ↗First posted 7 Feb 2025cs.LG cond-mat.dis-nn cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems. While these phenomena hold promise for uncovering the mechanisms behind neural networks and language models, existing theories remain fragmented, addressing specific cases. In this position paper, we advocate for the crucial role of the research direction of parameter symmetries in unifying these fragmented theories. This position is founded on a centralizing hypothesis for this direction: parameter symmetry breaking and restoration are the unifying mechanisms underlying the hierarchical learning behavior of AI models. We synthesize prior observations and theories to argue that this direction of research could lead to a unified understanding of three distinct hierarchies in neural networks: learning dynamics, model complexity, and representation formation. By connecting these hierarchies, our position paper elevates symmetry -- a cornerstone of theoretical physics -- to become a potential fundamental principle in modern AI.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 62 of its 87 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

62 of 87 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  8. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  9. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
38 more
  1. 2018On Lazy Training in Differentiable Programmingcited by 675
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2024The Platonic Representation Hypothesiscited by 382
  5. 2018Invariant and Equivariant Graph Networkscited by 346
  6. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  7. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  9. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  10. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  11. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  13. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  14. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  15. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  16. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  17. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  18. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  19. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  20. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  21. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 86
  22. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  23. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  24. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 64
  25. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  26. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  27. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  28. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  29. 2018Neural Network Renormalization Groupcited by 48
  30. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  31. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  32. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  33. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  34. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  35. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  36. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  37. 2024Weight decay induces low-rank attention layerscited by 27
  38. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  2. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 9
  3. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 2
  4. 2026PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priorscited by 1
  5. 2026Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Lawscited by 1
  6. 2025Intuition emerges in Maximum Caliber models at criticalitycited by 0
  7. 2025Fractal basin geometry and the limits to predictability and reproducibility in deep learningcited by 0
  8. 2025Symmetry and Generalisation in Neural Approximations of Renormalisation Transformationscited by 0
  9. 2025Noise-robust Contrastive Learning for Critical Transition Detection in Dynamical Systemscited by 0
  10. 2026Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weightscited by 0
  11. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0
  12. 2026Understanding Deep Learning via Entropy Space Theorycited by 0