mlsanityData to 14 Oct 2026

QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models

arXiv 2502.12346 ↗First posted 17 Feb 2025cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Language Models (LLMs) are often quantized to lower precision to reduce the memory cost and latency in inference. However, quantization often degrades model performance, thus fine-tuning is required for various down-stream tasks. Traditional fine-tuning methods such as stochastic gradient descent and Adam optimization require backpropagation, which are error-prone in the low-precision settings. To overcome these limitations, we propose the Quantized Zeroth-Order (QuZO) framework, specifically designed for fine-tuning LLMs through low-precision (e.g., 4- or 8-bit) forward passes. Our method can avoid the error-prone low-precision straight-through estimator, and utilizes optimized stochastic rounding to mitigate the increased bias. QuZO simplifies the training process, while achieving results comparable to first-order methods in rm FP8 and superior accuracy in rm INT8 and rm INT4 training. Experiments demonstrate that low-bit training QuZO achieves performance comparable to MeZO optimization on GLUE, Multi-Choice, and Generation tasks, while reducing memory cost by 2.94 × in LLaMA2-7B fine-tuning compared to quantized first-order methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

45 of 60 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
33 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2017Mixed Precision Trainingcited by 1,156
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  8. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  9. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2021A White Paper on Neural Network Quantizationcited by 472
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  13. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  14. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  15. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  16. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  17. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  18. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  19. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  20. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  21. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  22. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  23. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 79
  24. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  25. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  26. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  27. 2023Training Transformers with 4-bit Integerscited by 60
  28. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  29. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  30. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  31. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantizationcited by 20
  32. 2023Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformerscited by 18
  33. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokenscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025Fine-tuning Quantized Neural Networks with Zeroth-order Optimizationcited by 6
  2. 2026Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Costcited by 6
  3. 2025Improving the Straight-Through Estimator with Zeroth-Order Informationcited by 5
  4. 2025SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passescited by 4
  5. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Costcited by 3
  6. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  7. 2026Training Non-Differentiable Networks via Optimal Transportcited by 1
  8. 2026Fine-Tuning Low-Bit Models with Gradient in Quantized Code Spacecited by 1
  9. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 1
  10. 2026Hierarchical Zero-Order Optimization for Deep Neural Networkscited by 0
  11. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
  2. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
  3. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
  4. 2026ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
  5. 2024QEFT: Quantization for Efficient Fine-Tuning of LLMs
  6. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
  7. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  8. 2024Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models