mlsanityData to 14 Oct 2026

Inference-Native Zeroth-Order Optimization for LLMs

arXiv 2605.28760 ↗First posted 27 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) methods train large language models using only forward passes, yet common ZO execution paths perform substantial work beyond what the algorithm itself requires. To remove this extra execution overhead, we present Infer-ZO, which separates the evaluations required by the algorithm from how they are executed, allowing them to reuse existing inference-engine optimizations. After removing inherited execution work, a complete Infer-ZO step on Qwen3-14B adds only 1.16% wall-clock time over matched inference, bringing ZO execution close to its fundamental inference workload. With Infer-ZO, ZO evaluations run at near-inference cost with frozen base weights and can share GPU batches with ordinary inference requests. In co-serving experiments, foreground and background throughput remain within 1-3% of the corresponding background-request baseline. Across 15 Qwen3, Llama, and OPT models, Infer-ZO achieves 2.10x-9.94x end-to-end step speedups over released LoZO. Code is available at https://github.com/playeriv65/zo-vllm.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

26 of 36 references.

References · 26

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  11. 2022FP8 Formats for Deep Learningcited by 196
  12. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
14 more
  1. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  2. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  3. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  4. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  5. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  6. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  7. 2025Evolution Strategies at the Hyperscalecited by 25
  8. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  9. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  10. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  11. 2025Zeroth-Order Optimization is Secretly Single-Step Policy Optimizationcited by 7
  12. 2025ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memorycited by 6
  13. 2024MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Enginescited by 3
  14. 2026AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Momentscited by 2

Built on by

Not cited yet.