mlsanityData to 7 Oct 2026

Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Sampling

arXiv 2003.13001 ↗First posted 29 Mar 2020math.OC cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using (possibly noisy) evaluations of the function. Such optimization is also called derivative-free, zeroth-order, or black-box optimization. We propose a new Zeroth-Order Regularized Optimization method, dubbed ZORO. When the underlying gradient is approximately sparse at an iterate, ZORO needs very few objective function evaluations to obtain a new iterate that decreases the objective function. We achieve this with an adaptive, randomized gradient estimator, followed by an inexact proximal-gradient scheme. Under a novel approximately sparse gradient assumption and various different convex settings, we show the (theoretical and empirical) convergence rate of ZORO is only logarithmically dependent on the problem dimension. Numerical experiments show that ZORO outperforms the existing methods with similar assumptions, on both synthetic and real datasets.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 6; 2023: 4; 2024: 9; 2025: 11; 2026: 9.

45 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  3. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  4. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  5. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  6. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  7. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  8. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  9. 2024Gradient Compressed Sensing: A Query-Efficient Gradient Estimator for High-Dimensional Zeroth-Order Optimizationcited by 12
  10. 2023Certified Zeroth-order Black-Box Defense with Robust UNet Denoisercited by 11
  11. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  12. 2025Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbationscited by 9
33 more
  1. 2022Stochastic Zeroth order Descent with Structured Directionscited by 8
  2. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 8
  3. 2021Curvature-Aware Derivative-Free Optimizationcited by 7
  4. 2022Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Boundscited by 7
  5. 2021Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithmcited by 6
  6. 2021Global optimization using random embeddingscited by 6
  7. 2025ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memorycited by 5
  8. 2025ZOQO: Zero-Order Quantized Optimizationcited by 4
  9. 2025The Multi-Query Paradox in Zeroth-Order Optimizationcited by 4
  10. 2022Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivitycited by 3
  11. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
  12. 2022Lazy Queries Can Reduce Variance in Zeroth-order Optimizationcited by 2
  13. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  14. 2026FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptationcited by 2
  15. 2021A Dimension-Insensitive Algorithm for Stochastic Zeroth-Order Optimizationcited by 1
  16. 2021Communication-Efficient Distributed SGD with Compressed Sensingcited by 1
  17. 2024Stochastic Two Points Method for Deep Model Zeroth-order Optimizationcited by 1
  18. 2024A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detectioncited by 1
  19. 2025Efficient Personalization of Quantized Diffusion Model without Backpropagationcited by 1
  20. 2025DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computingcited by 1
  21. 2026ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimizationcited by 1
  22. 2026Optimal low-rank stochastic gradient estimation for LLM trainingcited by 1
  23. 2022Convergence of Batch Updating Methods with Approximate Gradients and/or Noisy Measurements: Theory and Computational Resultscited by 0
  24. 2024CONGO: Compressive Online Gradient Optimizationcited by 0
  25. 2024Communication-Efficient Federated Learning over Wireless Channels via Gradient Sketchingcited by 0
  26. 2025Explainable Adversarial Attacks on Coarse-to-Fine Classifierscited by 0
  27. 2025A Structured Tour of Optimization with Finite Differencescited by 0
  28. 2026Guiding the Recommender: Information-Aware Auto-Bidding for Content Promotioncited by 0
  29. 2026Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimizationcited by 0
  30. 2026ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Trainingcited by 0
  31. 2026Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNscited by 0
  32. 2026EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Samplecited by 0
  33. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0