mlsanityData to 14 Oct 2026

Symmetry in Neural Network Parameter Spaces

arXiv 2506.13018 ↗First posted 16 Jun 2025cs.LG cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Modern deep learning models are highly overparameterized, resulting in large sets of parameter configurations that yield the same outputs. A significant portion of this redundancy is explained by symmetries in the parameter space--transformations that leave the network function unchanged. These symmetries shape the loss landscape and constrain learning dynamics, offering a new lens for understanding optimization, generalization, and model complexity that complements existing theory of deep learning. This survey provides an overview of parameter space symmetry. We summarize existing literature, uncover connections between symmetry and learning theory, and identify gaps and opportunities in this emerging field.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 117 of its 156 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

117 of 156 references.

References · 117

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2016Layer Normalizationcited by 5,669
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  6. 2018Group Normalizationcited by 1,524
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  9. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  10. 2016Group Equivariant Convolutional Networkscited by 1,165
  11. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  12. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
38 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  2. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  3. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  4. 2019General E(2)-Equivariant Steerable CNNscited by 446
  5. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  6. 2018Invariant and Equivariant Graph Networkscited by 346
  7. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  9. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  12. 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 194
  13. 2022From data to functa: Your data point is a function and you can treat it like onecited by 187
  14. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  15. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  16. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  17. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  18. 2017Equivariance Through Parameter-Sharingcited by 141
  19. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  20. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  21. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  22. 2020Predicting Neural Network Accuracy from Weightscited by 110
  23. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 107
  24. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  25. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  26. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  27. 2019Reverse-Engineering Deep ReLU Networkscited by 83
  28. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  29. 2023Permutation Equivariant Neural Functionalscited by 70
  30. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 66
  31. 2021Geometric Deep Learning and Equivariant Neural Networkscited by 65
  32. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 64
  33. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 63
  34. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  35. 2021A Survey of Complex-Valued Neural Networkscited by 61
  36. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  37. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 57
  38. 2023Neural Functional Transformerscited by 52

50 most cited of 117 All 117 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 29.

36 papers.

Sort:

By area:

  1. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  2. 2025Model Unmerging: Making Your Models Unmergeable for Secure Model Sharingcited by 6
  3. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  4. 2026Most ReLU Networks Admit Identifiable Parameterscited by 6
  5. 2026DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weightscited by 5
  6. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  7. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 4
  8. 2026NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spacescited by 3
  9. 2026A Complete Symmetry Classification of Shallow ReLU Networkscited by 3
  10. 2026The Symmetries of Three-Layer ReLU Networkscited by 3
  11. 2026Local Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and Human Braincited by 2
  12. 2026What Linear Probes Miss: Multi-View Probing for Weight-Space Learningcited by 2
24 more
  1. 2026On the Epistemic Uncertainty of Overparametrized Neural Networkscited by 2
  2. 2026Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformerscited by 2
  3. 2026Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networkscited by 2
  4. 2025Control Synthesis with Reinforcement Learning: A Modeling Perspectivecited by 1
  5. 2025Weight Space Representation Learning via Neural Field Adaptationcited by 1
  6. 2026Discovering Hidden Gems in Model Repositoriescited by 1
  7. 2026Complete Identification of Deep ReLU Networks through Lukasiewicz Logiccited by 1
  8. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 1
  9. 2026The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Samplingcited by 1
  10. 2026Dead Directions: Geometric Singular Learningcited by 1
  11. 2026PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priorscited by 1
  12. 2025Symmetry-Aware Fully-Amortized Optimization with Scale Equivariant Graph Metanetworkscited by 0
  13. 2025Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalizationcited by 0
  14. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  15. 2026Locking Pretrained Weights via Deep Low-Rank Residual Distillationcited by 0
  16. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  17. 2026Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weightscited by 0
  18. 2026Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memorycited by 0
  19. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0
  20. 2026Multi-Head Self Attention is a Parameter Identification Mechanismcited by 0
  21. 2026Discovering Symmetries in Neural Network Parameter Spacescited by 0
  22. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0
  23. 2026Training-Free Transformer Merging via Sequential Local Operator Alignmentcited by 0
  24. 2026Removing Information Content Does Not Certify Tamper Resistance in Open-Weight Modelscited by 0