mlsanityData to 14 Oct 2026

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

arXiv 2607.04033 ↗First posted 4 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 65 of its 139 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

65 of 139 references.

References · 65

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  9. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  12. 2024Old Optimizer, New Norm: An Anthologycited by 193
38 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  2. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  3. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  4. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  5. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  6. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
  7. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  8. 2025Dion: Distributed Orthonormalized Updatescited by 68
  9. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
  10. 2025AdaMuon: Adaptive Muon Optimizercited by 67
  11. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  12. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  13. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  14. 2022SAM as an Optimal Relaxation of Bayescited by 41
  15. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  16. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  17. 2018Aggregated Momentum: Stability Through Passive Dampingcited by 36
  18. 2026Controlled LLM Training on Spectral Spherecited by 31
  19. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  20. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  21. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  22. 2021Ranger21: a synergistic deep learning optimizercited by 22
  23. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 22
  24. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 15
  25. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
  26. 2019An Adaptive and Momental Bound Method for Stochastic Learningcited by 12
  27. 2020AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weightscited by 10
  28. 2025Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variantscited by 10
  29. 2025ZClip: Adaptive Spike Mitigation for LLM Pre-Trainingcited by 10
  30. 2019Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networkscited by 9
  31. 2022Black Box Lie Group Preconditioners for SGDcited by 9
  32. 2025Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extensioncited by 9
  33. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 8
  34. 2025FOCUS: First Order Concentrated Updating Schemecited by 8
  35. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  36. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 7
  37. 2025AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clippingcited by 6
  38. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.