mlsanityData to 7 Oct 2026

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

arXiv 2502.03304 ↗First posted 5 Feb 2025cs.LG cs.AI cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recently, zeroth-order (ZO) optimization stood out as a promising memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource-constrained scenarios. However, ZO method lags far behind FO method in both convergence speed and accuracy. To bridge the gap, we introduce a novel layer-wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. Aiming to resemble the learning capacity of FO method from the findings, we propose Divergence-driven Zeroth-Order (DiZO) optimization. DiZO conducts divergence-driven layer adaptation by incorporating projections to ZO updates, generating diverse-magnitude updates precisely scaled to layer-wise individual optimization needs. Our results demonstrate that DiZO significantly reduces the needed iterations for convergence without sacrificing throughput, cutting training GPU hours by up to 48% on various datasets. Moreover, DiZO consistently outperforms the representative ZO baselines in fine-tuning RoBERTa-large, OPT-series, and Llama-series on downstream tasks and, in some cases, even surpasses memory-intensive FO fine-tuning. Our code is released at https://github.com/Skilteee/DiZO.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

51 of 71 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  11. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  14. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  15. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  16. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  17. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  18. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  19. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  20. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  21. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  22. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  23. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  24. 2024AI and Memory Wallcited by 89
  25. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  26. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  27. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  28. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  29. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 51
  30. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  31. 2019Model Agnostic Contrastive Explanations for Structured Datacited by 41
  32. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  33. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  34. 2019Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspectivecited by 26
  35. 2020Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimizationcited by 24
  36. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 17
  37. 2020Learning to Guide Random Searchcited by 13
  38. 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memorycited by 24
  2. 2025Structured Agent Distillation for Large Language Modelcited by 11
  3. 2025FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speedcited by 11
  4. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  5. 2025Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Trainingcited by 9
  6. 2025KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuningcited by 4
  7. 2025Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearningcited by 4
  8. 2026CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuningcited by 3
  9. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  10. 2026When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Makingcited by 2
  11. 2026AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Momentscited by 2
  12. 2025TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platformcited by 1
5 more
  1. 2026Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Samplingcited by 1
  2. 2026Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMscited by 1
  3. 2026Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMscited by 1
  4. 2026CooperLLM: Cloud-Edge-End Cooperative Federated Fine-tuning for LLMs via ZOO-based Gradient Correctioncited by 0
  5. 2026Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theorycited by 0