mlsanityData to 7 Oct 2026

The Role of Symmetry in Optimizing Overparameterized Networks

arXiv 2604.25150 ↗First posted 28 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. We analyze weight-space symmetries in neural networks and show that overparameterization introduces additional symmetries that benefit optimization in two distinct ways. First, we prove that these symmetries act as a form of diagonal preconditioning on the Hessian, enabling the existence of better-conditioned minima within each equivalence class of functionally identical solutions. Second, we show that overparameterization increases the probability mass of global minima near typical initializations, making these favourable solutions more reachable. These results offer a potential link between loss landscape geometry and simplicity bias. Empirically, we observe wider networks have lower top eigenvalues, smaller condition numbers and faster convergence, matching our analysis. Our analysis provides a unified framework for understanding overparameterization and width growth as a geometric transformation of the loss landscape.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

34 of 45 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  10. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  11. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
22 more
  1. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  2. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  3. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  4. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  5. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  6. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  7. 2017The loss surface of deep and wide neural networkscited by 74
  8. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  9. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  10. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  11. 2023Hidden symmetries of ReLU networkscited by 38
  12. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  13. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  14. 2015On the Quality of the Initial Basin in Overspecified Neural Networkscited by 20
  15. 2017Exponentially vanishing sub-optimal local minima in multilayer neural networkscited by 19
  16. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  17. 2023Expand-and-Cluster: Parameter Recovery of Neural Networkscited by 15
  18. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 10
  19. 2017Projection Based Weight Normalization for Deep Neural Networkscited by 8
  20. 2023No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Pathscited by 7
  21. 2025Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticitycited by 5
  22. 2023The ω-Condition Number: Applications to Optimal Preconditioning and Low Rank Generalized Jacobian Updatingcited by 1

Built on by

Not cited yet.