mlsanityData to 14 Oct 2026

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

arXiv 2606.02857 ↗First posted 1 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation. We propose GRZO, a Group-Relative Zeroth-Order optimizer that draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. We prove that GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO. Across RoBERTa-large, Llama3-8B, and OPT-13B over multiple tasks, GRZO improves average accuracy on Llama3-8B by +3.0 over MeZO at 23% lower peak GPU memory; as a drop-in replacement for the MeZO core, it lifts sparse, low-rank, and quantized ZO variants by +6.0 on average.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

30 of 42 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  12. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
18 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  3. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  4. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  5. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  6. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  7. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  8. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 54
  9. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  10. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  11. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  12. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  13. 2025FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speedcited by 14
  14. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  15. 2025Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbationscited by 11
  16. 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
  17. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  18. 2025SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passescited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory
  2. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
  3. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
  4. 2026ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
  5. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  6. 2025Memory Savings at What Cost? A Study of Alternatives to Backpropagation
  7. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning
  8. 2024Scaling Sparse Fine-Tuning to Large Language Models