mlsanityData to 7 Oct 2026

Remove Symmetries to Control Model Expressivity and Improve Optimization

arXiv 2408.15495 ↗First posted 28 Aug 2024cs.LG cs.AI stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

When symmetry is present in the loss function, the model is likely to be trapped in a low-capacity state that is sometimes known as a "collapse". Being trapped in these low-capacity states can be a major obstacle to training across many scenarios where deep learning technology is applied. We first prove two concrete mechanisms through which symmetries lead to reduced capacities and ignored features during training and inference. We then propose a simple and theoretically justified algorithm, syre, to remove almost all symmetry-induced low-capacity states in neural networks. When this type of entrapment is especially a concern, removing symmetries with the proposed method is shown to correlate well with improved optimization or performance. A remarkable merit of the proposed method is that it is model-agnostic and does not require any knowledge of the symmetry.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

29 of 49 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2016Group Equivariant Convolutional Networkscited by 1,154
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
17 more
  1. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  2. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  3. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  4. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
  5. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  6. 2023Posterior Collapse and Latent Variable Non-identifiabilitycited by 57
  7. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  8. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  9. 2020Self-Supervised Relational Reasoning for Representation Learningcited by 37
  10. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  11. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  12. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  13. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  14. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  15. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  16. 2022spred: Solving L1 Penalty with SGDcited by 16
  17. 2021Smooth Bilevel Programming for Sparse Regularizationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Symmetry in Neural Network Parameter Spacescited by 31
  2. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  3. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  4. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 2
  5. 2026On the Epistemic Uncertainty of Overparametrized Neural Networkscited by 2
  6. 2025Topological Invariance and Breakdown in Learningcited by 1
  7. 2025Fractal basin geometry and the limits to predictability and reproducibility in deep learningcited by 0
  8. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0