mlsanityData to 7 Oct 2026

Improving Convergence and Generalization Using Parameter Symmetries

arXiv 2305.13404 ↗First posted 22 May 2023cs.LG math.OCcited by 23

Not on the map. Why?

Abstract (arXiv)

In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only speeds up optimization in the short-term, but gives overall faster time to convergence. Additionally, teleporting to minima with different curvatures improves generalization, which suggests a connection between the curvature of the minimum and generalization ability. Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence. Our results showcase the versatility of teleportation and demonstrate the potential of incorporating symmetry in optimization.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

40 of 51 references.

References · 40

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  7. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  8. 2016Learning to learn by gradient descent by gradient descentcited by 616
  9. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  12. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
28 more
  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  3. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  4. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  5. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  6. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  7. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  8. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  9. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  10. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  12. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  13. 2020Relative Flatness and Generalizationcited by 88
  14. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  15. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  16. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
  17. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  18. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  19. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 26
  20. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  21. 2022Functional dimension of feedforward ReLU neural networkscited by 20
  22. 2020The Representation Theory of Neural Networkscited by 18
  23. 2020Neural Teleportationcited by 17
  24. 2015Symmetry-invariant optimization in deep networkscited by 15
  25. 2021Universal approximation and model compression for radial neural networkscited by 7
  26. 2019On Symmetry and Initialization for Neural Networkscited by 5
  27. 2022How You Start Matters for Generalizationcited by 5
  28. 2022Quiver neural networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 11; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  2. 2025Symmetry in Neural Network Parameter Spacescited by 31
  3. 2024Scale Equivariant Graph Metanetworkscited by 23
  4. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  5. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  6. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  7. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  8. 2024Remove Symmetries to Control Model Expressivity and Improve Optimizationcited by 8
  9. 2025ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusioncited by 6
  10. 2025Continual Optimization with Symmetry Teleportation for Multi-Task Learningcited by 3
  11. 2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
  12. 2026DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weightscited by 3
11 more
  1. 2025Teleportation With Null Space Gradient Projection for Optimization Accelerationcited by 2
  2. 2025Improving Learning to Optimize Using Parameter Symmetriescited by 2
  3. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  4. 2024Level Set Teleportation: An Optimization Perspectivecited by 1
  5. 2024Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUscited by 1
  6. 2025WARP: Weight Teleportation for Attack-Resilient Unlearning Protocolscited by 1
  7. 2026PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priorscited by 1
  8. 2023Sharpness-Aware Teleportation on Riemannian Manifoldscited by 0
  9. 2026Optimizer-Induced Mode Connectivity: From AdamW to Muoncited by 0
  10. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  11. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0