mlsanityData to 14 Oct 2026

Optimistic Dual Averaging Unifies Modern Optimizers

arXiv 2605.11172 ↗First posted 11 May 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce SODA, a generalization of Optimistic Dual Averaging, which provides a common perspective on state-of-the-art optimizers like Muon, Lion, AdEMAMix and NAdam, showing that they can all be viewed as optimistic instances of this framework. Based on this framing, we propose a practical SODA wrapper for any base optimizer that eliminates weight decay tuning through a theoretically-grounded 1/k decay schedule. Empirical results across various scales and training horizons show that SODA consistently improves performance without any additional hyperparameter tuning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

27 of 48 references.

References · 27

  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  3. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  5. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  6. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  7. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  8. 2024The Road Less Scheduledcited by 163
  9. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  10. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  11. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
  12. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
15 more
  1. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  2. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  3. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  4. 2018Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximizationcited by 47
  5. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  6. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 28
  7. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  8. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  9. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  10. 2025Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variantscited by 10
  11. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  12. 2026Logarithmic-time Schedules for Scaling Language Models with Momentumcited by 9
  13. 2025SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradientscited by 8
  14. 2020Dual Averaging is Surprisingly Effective for Deep Learning Optimizationcited by 6
  15. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When to use what Schatten-p norm in deep learning?cited by 2
  2. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  3. 2026AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Modelscited by 0