mlsanityData to 7 Oct 2026

A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning

arXiv 2006.06224 ↗First posted 11 Jun 2020cs.LG eess.SP stat.MLcited by 160

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization is a subset of gradient-free optimization that emerges in many signal processing and machine learning applications. It is used for solving optimization problems similarly to gradient-based methods. However, it does not require the gradient, using only function evaluations. Specifically, ZO optimization iteratively performs three major steps: gradient estimation, descent direction computation, and solution update. In this paper, we provide a comprehensive review of ZO optimization, with an emphasis on showing the underlying intuition, optimization principles and recent advances in convergence analysis. Moreover, we demonstrate promising applications of ZO optimization, such as evaluating robustness and generating explanations from black-box deep learning models, and efficient online sensor management.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 21; 2023: 24; 2024: 32; 2025: 44; 2026: 25.

160 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  3. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  4. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  5. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  6. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  7. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 63
  8. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  9. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  10. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  11. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  12. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
38 more
  1. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  2. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  3. 2021Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvaturecited by 32
  4. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 27
  5. 2021Link Scheduling using Graph Neural Networkscited by 26
  6. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  7. 2022Learning Physics-Informed Neural Networks without Stacked Back-propagationcited by 23
  8. 2021Zeroth-Order Methods for Convex-Concave Minmax Problems: Applications to Decision-Dependent Risk Minimizationcited by 18
  9. 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
  10. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
  11. 2026SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibecited by 18
  12. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 17
  13. 2024Decision-Focused Learning with Directional Gradientscited by 16
  14. 2023A survey and taxonomy of loss functions in machine learningcited by 15
  15. 2021Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGDcited by 14
  16. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  17. 2022Permutation Search of Tensor Network Structures via Local Samplingcited by 13
  18. 2023Overload: Latency Attacks on Object Detection for Edge Devicescited by 13
  19. 2023BAFFLE: A Baseline of Backpropagation-Free Federated Learningcited by 12
  20. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
  21. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  22. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 12
  23. 2022Holistic Adversarial Robustness of Deep Learning Modelscited by 11
  24. 2023Certified Zeroth-order Black-Box Defense with Robust UNet Denoisercited by 11
  25. 2023Performative Prediction with Bandit Feedback: Learning through Reparameterizationcited by 11
  26. 2024Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimizationcited by 11
  27. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  28. 2021Certifiably-Robust Federated Adversarial Learning via Randomized Smoothingcited by 9
  29. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  30. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  31. 2025Towards Effective and Sparse Adversarial Attack on Spiking Neural Networks via Breaking Invisible Surrogate Gradientscited by 9
  32. 2025Test-Time Model Adaptation for Quantized Neural Networkscited by 9
  33. 2025NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagationcited by 8
  34. 2021Derivative-free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problemscited by 7
  35. 2021Curvature-Aware Derivative-Free Optimizationcited by 7
  36. 2022Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Boundscited by 7
  37. 2025Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approachcited by 7
  38. 2021Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithmcited by 6

50 most cited of 160 All 160 (JSON)