mlsanityData to 7 Oct 2026

Old Optimizer, New Norm: An Anthology

arXiv 2409.20325 ↗First posted 30 Sep 2024cs.LG math.OCcited by 173

Not on the map. Why?

Abstract (arXiv)

Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods -- Adam, Shampoo and Prodigy -- and argue that each method can instead be understood as a squarely first-order method without convexity assumptions. In fact, after switching off exponential moving averages, each method is equivalent to steepest descent under a particular norm. By generalizing this observation, we chart a new design space for training algorithms. Different operator norms should be assigned to different tensors based on the role that the tensor plays within the network. For example, while linear and embedding layers may have the same weight space of ℝm× n, these layers play different roles and should be assigned different norms. We hope that this idea of carefully metrizing the neural architecture might lead to more stable, scalable and indeed faster training.

Key facts. Cited by 173 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

21 of 37 references.

References · 21

  1. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  4. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  5. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  6. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  7. 2023A Spectral Condition for Feature Learningcited by 101
  8. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  9. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  10. 2024Scalable Optimization in the Modular Normcited by 64
  11. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  12. 2024A New Perspective on Shampoo's Preconditionercited by 54
9 more
  1. 2023Benchmarking Neural Network Training Algorithmscited by 46
  2. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  3. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  4. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  5. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 16
  6. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  7. 2023Universal Majorization-Minimization Algorithmscited by 3
  8. 2017Unifying the Stochastic Spectral Descent for Restricted Boltzmann Machines with Bernoulli or Gaussian Inputscited by 1
  9. 2024Improving Line Search Methods for Large Scale Neural Network Trainingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 56; 2026: 112.

173 papers.

Sort:

By area:

  1. 2025Muon is Scalable for LLM Trainingcited by 381
  2. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  3. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  4. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  5. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  6. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  7. 2024Modular Duality in Deep Learningcited by 71
  8. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  9. 2025Dion: Distributed Orthonormalized Updatescited by 64
  10. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  11. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  12. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
38 more
  1. 2025In Search of Adam's Secret Saucecited by 40
  2. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  3. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  4. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  5. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  6. 2026Convergence of Muon with Newton-Schulzcited by 30
  7. 2025On the Convergence of Muon and Beyondcited by 28
  8. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  9. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 24
  10. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  11. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  12. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  13. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  14. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 20
  15. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  16. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  17. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  18. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  19. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  20. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  21. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 15
  22. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  23. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15
  24. 2026The Newton-Muon Optimizercited by 14
  25. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  26. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  27. 2025ROOT: Robust Orthogonalized Optimizer for Neural Network Trainingcited by 13
  28. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 13
  29. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  30. 2025Cautious Weight Decaycited by 12
  31. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  32. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 11
  33. 2024DeMo: Decoupled Momentum Optimizationcited by 10
  34. 2025Effective Quantization of Muon Optimizer Statescited by 10
  35. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  36. 2026FISMO: Fisher-Structured Momentum-Orthogonalized Optimizercited by 10
  37. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  38. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9

50 most cited of 173 All 173 (JSON)