mlsanityData to 14 Oct 2026

Stochastic Zeroth-order Optimization in High Dimensions

arXiv 1710.10551 ↗First posted 29 Oct 2017stat.ML cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

We consider the problem of optimizing a high-dimensional convex function using stochastic zeroth-order queries. Under sparsity assumptions on the gradients or function values, we present two algorithms: a successive component/feature selection algorithm and a noisy mirror descent algorithm using Lasso gradient estimates, and show that both algorithms have convergence rates that de- pend only logarithmically on the ambient dimension of the problem. Empirical results confirm our theoretical findings and show that the algorithms we design outperform classical zeroth-order optimization methods in the high-dimensional setting.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 8; 2023: 10; 2024: 9; 2025: 13; 2026: 13.

63 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  2. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  3. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  4. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  5. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 54
  6. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  7. 2022Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimizationcited by 43
  8. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  9. 2023Gradient-Free Textual Inversioncited by 29
  10. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  11. 2023Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimizationcited by 17
  12. 2021Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methodscited by 14
38 more
  1. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  2. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  3. 2024Gradient Compressed Sensing: A Query-Efficient Gradient Estimator for High-Dimensional Zeroth-Order Optimizationcited by 12
  4. 2022On Adaptivity in Non-stationary Stochastic Optimization With Bandit Feedbackcited by 11
  5. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  6. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 9
  7. 2021Curvature-Aware Derivative-Free Optimizationcited by 7
  8. 2022Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Boundscited by 7
  9. 2021Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithmcited by 6
  10. 2021Global optimization using random embeddingscited by 6
  11. 2025Boosting Offline Optimizers with Surrogate Sensitivitycited by 6
  12. 2021Problem Dependent View on Structured Thresholding Bandit Problemscited by 5
  13. 2026Improved Dimension Dependence for Bandit Convex Optimization with Gradient Variationscited by 5
  14. 2021SGD with Coordinate Sampling: Theory and Practicecited by 4
  15. 2024Covariance-Adaptive Sequential Black-box Optimization for Diffusion Targeted Generationcited by 4
  16. 2025ZOQO: Zero-Order Quantized Optimizationcited by 4
  17. 2025Dynamic Pricing with Adversarially-Censored Demandscited by 4
  18. 2025The Multi-Query Paradox in Zeroth-Order Optimizationcited by 4
  19. 2025ROOT: Rethinking Offline Optimization as Distributional Translation via Probabilistic Bridgecited by 4
  20. 2022Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivitycited by 3
  21. 2023Step by Step Loss Goes Very Far: Multi-Step Quantization for Adversarial Text Attackscited by 3
  22. 2026FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptationcited by 3
  23. 2022Lazy Queries Can Reduce Variance in Zeroth-order Optimizationcited by 2
  24. 2024Forward Learning for Gradient-based Black-box Saliency Map Generationcited by 2
  25. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  26. 2021A Dimension-Insensitive Algorithm for Stochastic Zeroth-Order Optimizationcited by 1
  27. 2022Adaptive Zeroth-Order Optimisation of Nonconvex Composite Objectivescited by 1
  28. 2024Stochastic Two Points Method for Deep Model Zeroth-order Optimizationcited by 1
  29. 2024Gradient-Free Method for Heavily Constrained Nonconvex Optimizationcited by 1
  30. 2025VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergencecited by 1
  31. 2025Memory Savings at What Cost? A Study of Alternatives to Backpropagationcited by 1
  32. 2021Scalable Projection-Free Optimizationcited by 0
  33. 2022Convergence of Batch Updating Methods with Approximate Gradients and/or Noisy Measurements: Theory and Computational Resultscited by 0
  34. 2022Adaptive Stochastic Optimisation of Nonconvex Composite Objectivescited by 0
  35. 2023Reward Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Lawscited by 0
  36. 2024CONGO: Compressive Online Gradient Optimizationcited by 0
  37. 2025Mirror Descent Linearized Augmented Lagrangian Methods for Nonconvex Constrained Stochastic Zeroth-Order Optimizationcited by 0
  38. 2025Estimating the Effects of Sample Training Orders for Large Language Models without Retrainingcited by 0

50 most cited of 63 All 63 (JSON)