mlsanityData to 14 Oct 2026

Improving Learning to Optimize Using Parameter Symmetries

arXiv 2504.15399 ↗First posted 21 Apr 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance optimization efficiency. Prior work has shown that jointly learning symmetry transformations and local updates improves meta-optimizer performance. Supporting this, our theoretical analysis demonstrates that even without identifying the optimal group element, the method locally resembles Newton's method. We further provide an example where the algorithm provably learns the correct symmetry transformation during training. To empirically evaluate L2O with teleportation, we introduce a benchmark, analyze its success and failure cases, and show that enhancements like momentum further improve performance. Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

30 of 38 references.

References · 30

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2016Learning to learn by gradient descent by gradient descentcited by 620
  3. 2019Meta-Learning with Implicit Gradientscited by 525
  4. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 379
  5. 2021Learning to Optimize: A Primer and A Benchmarkcited by 208
  6. 2016Learning to Optimizecited by 195
  7. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  8. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  9. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  10. 2017Learned Optimizers that Scale and Generalizecited by 110
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  12. 2023Permutation Equivariant Neural Functionalscited by 70
18 more
  1. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  2. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 57
  3. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  4. 2017Learning to Optimize Neural Netscited by 41
  5. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 27
  6. 2024Universal Neural Functionalscited by 26
  7. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  8. 2024Scale Equivariant Graph Metanetworkscited by 24
  9. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  10. 2020Reverse engineering learned optimizers reveals known and novel mechanismscited by 19
  11. 2020Neural Teleportationcited by 18
  12. 2018Accelerating Natural Gradient with Higher-Order Invariancecited by 17
  13. 2019Equi-normalization of Neural Networkscited by 16
  14. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  15. 2015Symmetry-invariant optimization in deep networkscited by 15
  16. 2024Equivariant Polynomial Functional Networkscited by 11
  17. 2021Accelerating Training of Batch Normalization: A Manifold Perspectivecited by 4
  18. 2022Learning to Optimize Quasi-Newton Methodscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Symmetry in Neural Network Parameter Spacescited by 36
  2. 2026Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalizationcited by 1