mlsanityData to 7 Oct 2026

CAME: Confidence-guided Adaptive Memory Efficient Optimization

arXiv 2307.02047 ↗First posted 5 Jul 2023cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models. Nevertheless, the need for adaptivity requires maintaining second-moment estimates of the per-parameter gradients, which entails a high cost of extra memory overheads. To solve this problem, several memory-efficient optimizers (e.g., Adafactor) have been proposed to obtain a drastic reduction in auxiliary memory usage, but with a performance penalty. In this paper, we first study a confidence-guided strategy to reduce the instability of existing memory efficient optimizers. Based on this strategy, we propose CAME to simultaneously achieve two goals: fast convergence as in traditional adaptive methods, and low memory usage as in memory-efficient methods. Extensive experiments demonstrate the training stability and superior performance of CAME across various NLP tasks such as BERT and GPT-2 training. Notably, for BERT pre-training on the large batch size of 32,768, our proposed optimizer attains faster convergence and higher accuracy compared with the Adam optimizer. The implementation of CAME is publicly available.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

24 of 32 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
12 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  4. 2017Large Batch Training of Convolutional Networkscited by 529
  5. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  6. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  7. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  8. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  9. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  10. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  11. 2019Memory-Efficient Adaptive Optimizationcited by 19
  12. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 19; 2026: 6.

34 papers.

Sort:

By area:

  1. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  2. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  4. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  5. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  6. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  7. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  8. 2023KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesiscited by 18
  9. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  10. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 8
  11. 2024Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matricescited by 7
  12. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
22 more
  1. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  2. 2025Jodi: Unification of Visual Generation and Understanding via Joint Modelingcited by 5
  3. 2025A Training-free Method for LLM Text Attributioncited by 4
  4. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-trainingcited by 4
  5. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  6. 2025MobileI2V: Fast and High-Resolution Image-to-Video on Mobile Devicescited by 4
  7. 2025A Survey on Memory-Efficient Transformer-Based Model Training in AI for Sciencecited by 3
  8. 2025When Can You Get Away with Low Memory Adam?cited by 3
  9. 2025AdaPM: a Partial Momentum Algorithm for LLM Trainingcited by 3
  10. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  11. 2023LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Modelscited by 2
  12. 2025Taming LLMs by Scaling Learning Rates with Gradient Groupingcited by 1
  13. 2025Sparse Fine-Tuning of Transformers for Generative Taskscited by 1
  14. 2025MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Trainingcited by 0
  15. 2025LinearSR: Unlocking Linear Attention for Stable and Efficient Image Super-Resolutioncited by 0
  16. 2025Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimizationcited by 0
  17. 2025Bridging Training and Merging Through Momentum-Aware Optimizationcited by 0
  18. 2026SANA I2I: A Text Free Flow Matching Framework for Paired Image to Image Translation with a Case Study in Fetal MRI Artifact Reductioncited by 0
  19. 2026MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimizationcited by 0
  20. 2026Training Variable Long Sequences with Data-Centric Parallelcited by 0
  21. 2026Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizerscited by 0
  22. 2026TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuningcited by 0