mlsanityData to 7 Oct 2026

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

arXiv 2605.15622 ↗First posted 15 May 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 75 of its 108 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

75 of 108 references.

References · 75

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
38 more
  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  3. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  6. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  8. 2025Muon is Scalable for LLM Trainingcited by 381
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  10. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  11. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  12. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  13. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  14. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  15. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
  16. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  17. 2019Training Neural Networks with Local Error Signalscited by 135
  18. 2024Barren Plateaus in Variational Quantum Computingcited by 130
  19. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  20. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  21. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  22. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  23. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  24. 2022Gradients without Backpropagationcited by 73
  25. 2022Scaling Forward Gradient With Local Lossescited by 65
  26. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  27. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  28. 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
  29. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  30. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 51
  31. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  32. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  33. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  34. 2023Zero Bubble Pipeline Parallelismcited by 39
  35. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  36. 2023On quantum backpropagation, information reuse, and cheating measurement collapsecited by 32
  37. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  38. 2020Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimizationcited by 24

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspacescited by 0