mlsanityData to 14 Oct 2026

Cautious Optimizers: Improving Training with One Line of Code

arXiv 2411.16085 ↗First posted 25 Nov 2024cs.LG cs.AI cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a one-line modification in Pytorch to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters. Code is available at https://github.com/kyleliang919/C-Optim.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

42 of 62 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
30 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  13. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  14. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  15. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  16. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  17. 2024The Road Less Scheduledcited by 163
  18. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  19. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  20. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  21. 2013Hamiltonian Monte Carlo for Hierarchical Modelscited by 52
  22. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  23. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
  24. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  25. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  26. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  27. 2024ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Ratecited by 18
  28. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  29. 2023Variational Principles for Mirror Descent and Mirror Langevin Dynamicscited by 5
  30. 2021Extending AdamW by Leveraging Its Second Moment and Magnitudecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 27.

47 papers.

Sort:

By area:

  1. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  2. 2025On the Convergence of Muon and Beyondcited by 31
  3. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 16
  4. 2026Training-Free Looped Transformerscited by 15
  5. 2025Cautious Weight Decaycited by 13
  6. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  7. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  8. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  9. 2024DeMo: Decoupled Momentum Optimizationcited by 11
  10. 2025MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidancecited by 9
  11. 2026Delving into Muon and Beyond: Deep Analysis and Extensionscited by 9
  12. 2025FOCUS: First Order Concentrated Updating Schemecited by 8
35 more
  1. 2025Physics of Skill Learningcited by 8
  2. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 8
  3. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  4. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  5. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  6. 2025Relative Scaling Laws for LLMscited by 6
  7. 2025InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusioncited by 5
  8. 2025Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learningcited by 5
  9. 2026Benchmarking Optimizers for MLPs in Tabular Deep Learningcited by 5
  10. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  11. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  12. 2024Grams: Gradient Descent with Adaptive Momentum Scalingcited by 4
  13. 2026Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minimacited by 4
  14. 2026Momentum Guidance: Plug-and-Play Guidance for Flow Modelscited by 3
  15. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  16. 2026SAME: A Semantically-Aligned Music Autoencodercited by 3
  17. 2024CAdam: Confidence-Based Optimization for Online Learningcited by 2
  18. 2025Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rulescited by 2
  19. 2025Greedy Alignment Principle for Optimizer Selectioncited by 2
  20. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  21. 2025JAX-IK: Real-Time Inverse Kinematics for Generating Multi-Constrained Movements of Virtual Human Characterscited by 1
  22. 2025ViewSparsifier: Killing Redundancy in Multi-View Plant Phenotypingcited by 1
  23. 2025Algorithms Trained on Normal Chest X-rays Can Predict Health Insurance Typescited by 1
  24. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  25. 2026Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretrainingcited by 0
  26. 2026FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneitycited by 0
  27. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  28. 2026LionVote: Per-Layer Learning Rate Adaptation for Lioncited by 0
  29. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0
  30. 2026Blog: Survey of Optimizerscited by 0
  31. 2026Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Trainingcited by 0
  32. 2026Physical Muon: Orthogonalization as an Equilibrium Computationcited by 0
  33. 2026A Step Towards Forgetting: Optimiser History and the Loss of Answer Masscited by 0
  34. 2026ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Trainingcited by 0
  35. 2026Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerfcited by 0