Blog: Survey of Optimizers

arXiv 2608.28557 ↗First posted 28 Aug 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

52 of 70 references.

References · 52

2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
2023Symbolic Discovery of Optimization Algorithmscited by 434
2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
20218-bit Optimizers via Block-wise Quantizationcited by 403
2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
2024The Road Less Scheduledcited by 163
2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137

38 more

2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
2025NorMuon: Making Muon more efficient and scalablecited by 84
2025Dion: Distributed Orthonormalized Updatescited by 68
2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
2025AdaMuon: Adaptive Muon Optimizercited by 67
2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 28
2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 18
2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 17
2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
2025What Really Matters in Matrix-Whitening Optimizers?cited by 16
2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 14
2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
2025Cautious Weight Decaycited by 13
2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
2025Effective Quantization of Muon Optimizer Statescited by 11
2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 10
2025Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extensioncited by 9
2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 7
2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
2026ECO: Quantized Training without Full-Precision Master Weightscited by 5
2026STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Trainingcited by 4
2026Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handlingcited by 4
2025When Can You Get Away with Low Memory Adam?cited by 3
2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way