mlsanityData to 14 Oct 2026

A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization

arXiv 1905.01332 ↗First posted 3 May 2019math.OCcited by 68

Not on the map. Why?

Abstract (arXiv)

In this paper, we analyze several methods for approximating gradients of noisy functions using only function values. These methods include finite differences, linear interpolation, Gaussian smoothing and smoothing on a sphere. The methods differ in the number of functions sampled, the choice of the sample points, and the way in which the gradient approximations are derived. For each method, we derive bounds on the number of samples and the sampling radius which guarantee favorable convergence properties for a line search or fixed step size descent method. To this end, we use the results in [Berahas et al., 2019] and show how each method can satisfy the sufficient conditions, possibly only with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a sphere. Finally, we present numerical results evaluating the quality of the gradient approximations as well as their performance in conjunction with a line search derivative-free optimization algorithm.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 10; 2024: 15; 2025: 18; 2026: 10.

68 papers.

Sort:

By area:

  1. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  2. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  3. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 74
  4. 2025InverseBench: Benchmarking Plug-and-Play Diffusion Priors for Inverse Problems in Physical Sciencescited by 68
  5. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  6. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  7. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  8. 2021On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithmscited by 23
  9. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulationcited by 20
  10. 2024Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problemscited by 19
  11. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 18
  12. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
38 more
  1. 2022DNNR: Differential Nearest Neighbors Regressioncited by 11
  2. 2022Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learningcited by 11
  3. 2025Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbationscited by 11
  4. 2024Group Distributionally Robust Dataset Distillation with Risk Minimizationcited by 10
  5. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  6. 2022Stochastic Zeroth order Descent with Structured Directionscited by 8
  7. 2023Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradientscited by 8
  8. 2021Derivative-free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problemscited by 7
  9. 2021Curvature-Aware Derivative-Free Optimizationcited by 7
  10. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  11. 2024Ferret: Federated Full-Parameter Tuning at Scale for Large Language Modelscited by 6
  12. 2025Graph Guided Diffusion: Unified Guidance for Conditional Graph Generationcited by 5
  13. 2021Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimizationcited by 4
  14. 2022Model Calibration of the Liquid Mercury Spallation Target using Evolutionary Neural Networks and Sparse Polynomial Expansionscited by 4
  15. 2023FedZeN: Towards superlinear zeroth-order federated learning via incremental Hessian estimationcited by 4
  16. 2024Generalizing Stochastic Smoothing for Differentiation and Gradient Estimationcited by 4
  17. 2024Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learningcited by 3
  18. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
  19. 2024Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimizationcited by 2
  20. 2024Generalized Bayesian deep reinforcement learningcited by 2
  21. 2025Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scalecited by 2
  22. 2025Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networkscited by 2
  23. 2025Subspace-based Approximate Hessian Method for Zeroth-Order Optimizationcited by 2
  24. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  25. 2026Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An α-Potential Approachcited by 2
  26. 2026On Adaptivity in Zeroth-Order Optimizationcited by 2
  27. 2022Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problemscited by 1
  28. 2022Finding mixed-strategy equilibria of continuous-action games without gradients using randomized policy networkscited by 1
  29. 2023ApproxED: Approximate exploitability descent via learned best responsescited by 1
  30. 2023The limitation of neural nets for approximation and optimizationcited by 1
  31. 2023Unsupervised Random Quantum Networks for PDEscited by 1
  32. 2024End-to-End Mesh Optimization of a Hybrid Deep Learning Black-Box PDE Solvercited by 1
  33. 2024A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detectioncited by 1
  34. 2024Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradientcited by 1
  35. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradientscited by 1
  36. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Gamescited by 1
  37. 2025SEGA: A Transferable Signed Ensemble Gaussian Black-Box Attack against No-Reference Image Quality Assessment Modelscited by 1
  38. 2025Adversarial Attacks on Downstream Weather Forecasting Models: Application to Tropical Cyclone Trajectory Predictioncited by 1

50 most cited of 68 All 68 (JSON)