mlsanityData to 7 Oct 2026

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity

arXiv 2406.02913 ↗First posted 5 Jun 2024cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, the application of ZO fine-tuning in memory-constrained settings such as mobile phones and laptops is still challenging since full precision forward passes are infeasible. In this study, we address this limitation by integrating sparsity and quantization into ZO fine-tuning of LLMs. Specifically, we investigate the feasibility of fine-tuning an extremely small subset of LLM parameters using ZO. This approach allows the majority of un-tuned parameters to be quantized to accommodate the constraint of limited device memory. Our findings reveal that the pre-training process can identify a set of "sensitive parameters" that can guide the ZO fine-tuning of LLMs on downstream tasks. Our results demonstrate that fine-tuning 0.1% sensitive parameters in the LLM with ZO can outperform the full ZO fine-tuning performance, while offering wall-clock time speedup. Additionally, we show that ZO fine-tuning targeting these 0.1% sensitive parameters, combined with 4 bit quantization, enables efficient ZO fine-tuning of an Llama2-7B model on a GPU device with less than 8 GiB of memory and notably reduced latency.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

39 of 55 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023Mistral 7Bcited by 3,065
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
27 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  7. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  8. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  11. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  12. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
  13. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  14. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  15. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  16. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  17. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
  18. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  19. 2023Training Transformers with 4-bit Integerscited by 58
  20. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  21. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 51
  22. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 22
  23. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  24. 2018Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attackcited by 14
  25. 2023Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Interventioncited by 12
  26. 2023Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Modelcited by 9
  27. 2020Sparse Perturbations for Improved Convergence in Stochastic Zeroth-Order Optimizationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 6.

27 papers.

Sort:

By area:

  1. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  2. 2024Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixturescited by 12
  3. 2024Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Modelscited by 11
  4. 2025FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speedcited by 11
  5. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  6. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  7. 2025Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbationscited by 9
  8. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 8
  9. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  10. 2025Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Trainingcited by 5
  11. 2025Fine-tuning Quantized Neural Networks with Zeroth-order Optimizationcited by 5
  12. 2025ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimizationcited by 4
15 more
  1. 2025ZOQO: Zero-Order Quantized Optimizationcited by 4
  2. 2025Learning a Zeroth-Order Optimizer for Fine-Tuning LLMscited by 3
  3. 2025Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scalecited by 2
  4. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  5. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  6. 2026AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Momentscited by 2
  7. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruningcited by 1
  8. 2025Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Modelscited by 1
  9. 2025High-Layer Attention Pruning with Rescalingcited by 1
  10. 2025Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Modelscited by 0
  11. 2025On the Inherent Privacy of Zeroth Order Projected Gradient Descentcited by 0
  12. 2026Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuningcited by 0
  13. 2026New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Frameworkcited by 0
  14. 2026Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptationcited by 0
  15. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 0