mlsanityData to 14 Oct 2026

Symmetry Teleportation for Accelerated Optimization

arXiv 2205.10637 ↗First posted 21 May 2022cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove a necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad for several optimization problems including test functions, multi-layer regressions, and MNIST classification.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

20 of 33 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  7. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  8. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  9. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  10. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  12. 2013A Literature Survey of Benchmark Functions For Global Optimization Problemscited by 77
8 more
  1. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 27
  2. 2020The Representation Theory of Neural Networkscited by 19
  3. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  4. 2020Neural Teleportationcited by 18
  5. 2015Symmetry-invariant optimization in deep networkscited by 15
  6. 2018Improving Optimization for Models With Continuous Symmetry Breakingcited by 8
  7. 2019Lecture Notes: Optimization for Machine Learningcited by 8
  8. 2021Universal approximation and model compression for radial neural networkscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 5; 2025: 10; 2026: 4.

25 papers.

Sort:

By area:

  1. 2023Hidden symmetries of ReLU networkscited by 39
  2. 2025Symmetry in Neural Network Parameter Spacescited by 36
  3. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  4. 2024Scale Equivariant Graph Metanetworkscited by 24
  5. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  6. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  7. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  8. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  9. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  10. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  11. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 9
  12. 2023rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learningcited by 6
13 more
  1. 2025Continual Optimization with Symmetry Teleportation for Multi-Task Learningcited by 3
  2. 2025Teleportation With Null Space Gradient Projection for Optimization Accelerationcited by 2
  3. 2025Improving Learning to Optimize Using Parameter Symmetriescited by 2
  4. 2024Level Set Teleportation: An Optimization Perspectivecited by 1
  5. 2025WARP: Weight Teleportation for Attack-Resilient Unlearning Protocolscited by 1
  6. 2023Reflection Invariance Learning for Few-shot Semantic Segmentationcited by 0
  7. 2023Investigation into the Training Dynamics of Learned Optimizerscited by 0
  8. 2024A Theory of Synaptic Neural Balance: From Local to Global Ordercited by 0
  9. 2025Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliancecited by 0
  10. 2026Exploiting weight-space symmetries for approximating curvaturecited by 0
  11. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  12. 2026Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networkscited by 0
  13. 2026Discovering Symmetries in Neural Network Parameter Spacescited by 0