mlsanityData to 14 Oct 2026

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments

arXiv 2605.00650 ↗First posted 1 May 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Fine-tuning LLMs is necessary for various dedicated downstream tasks, but classic backpropagation-based fine-tuning methods require substantial GPU memory. To this end, a recent work, MeZO, which relies solely on forward passes to fine-tune LLMs, significantly reduces GPU requirements at the cost of slower convergence due to its indifference to loss landscapes. Standard solutions, such as Adam, explore loss landscapes by estimating the first- and second-order moments and storing them in memory to guide the model's movement through dimensions with lower curvature and vice versa. However, directly applying Adam negates MeZO's advantage as it will triple the memory requirement. In light of this, we propose AdaMeZO, a zeroth-order optimizer that leverages Adam-style first- and second-moment estimates without maintaining them in memory. We present a theoretical analysis of AdaMeZO, corroborated by extensive experiments demonstrating AdaMeZO's performance, showing that AdaMeZO can outperform MeZO while requiring up to 70% fewer forward passes. Trajectory visualizations affirm AdaMeZO's ability to adapt to diverse loss landscapes.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

33 of 59 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
21 more
  1. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  2. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  3. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  4. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  5. 2024The Road Less Scheduledcited by 163
  6. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  7. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  8. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  9. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  10. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  11. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  12. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  13. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  14. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  15. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  16. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35
  17. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 28
  18. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 25
  19. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  20. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 18
  21. 2025A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 1
  2. 2026Inference-Native Zeroth-Order Optimization for LLMscited by 0