# Derivative-free optimization methods

arXiv: https://arxiv.org/abs/1904.11585 · HTML: https://mlsanity.com/paper/1904.11585/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 6 of 6)

- [Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.16371/) (2025)
- [No-Regret Constrained Bayesian Optimization of Noisy and Expensive Hybrid Models using Differentiable Quantile Function Approximations](https://mlsanity.com/paper/2305.03824/) (2023)
- [Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimization](https://mlsanity.com/paper/2109.12213/) (2021)
- [Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization](https://mlsanity.com/paper/2602.13513/) (2026)
- [Extremization to Fine Tune Physics Informed Neural Networks for Solving Boundary Value Problems](https://mlsanity.com/paper/2406.05290/) (2024)
- [Comparison of derivative-free and gradient-based minimization for multi-objective compositional design of shape memory alloys](https://mlsanity.com/paper/2508.14127/) (2025)

## Abstract (arXiv)

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.