# Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers

arXiv: https://arxiv.org/abs/1804.00154 · HTML: https://mlsanity.com/paper/1804.00154/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2018.

## Built on by (most cited first, 9 of 9)

- [Arbitrage-free neural-SDE market models](https://mlsanity.com/paper/2105.11053/) (2021)
- [No-Regret Constrained Bayesian Optimization of Noisy and Expensive Hybrid Models using Differentiable Quantile Function Approximations](https://mlsanity.com/paper/2305.03824/) (2023)
- [Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System](https://mlsanity.com/paper/2110.11130/) (2021)
- [An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning](https://mlsanity.com/paper/2308.10098/) (2023)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Datasets and Benchmarks for Nanophotonic Structure and Parametric Design Simulations](https://mlsanity.com/paper/2310.19053/) (2023)
- [Interpretable Modeling and Reduction of Unknown Errors in Mechanistic Operators](https://mlsanity.com/paper/2211.01373/) (2022)
- [Geoff: The Generic Optimization Framework & Frontend for Particle Accelerator Controls](https://mlsanity.com/paper/2506.03796/) (2025)
- [Posterior Inference of Hamiltonian Parameters from RIXS Spectroscopy](https://mlsanity.com/paper/2608.13848/) (2026)

## Abstract (arXiv)

We present DFO-LS, a software package for derivative-free optimization (DFO) for nonlinear Least-Squares (LS) problems, with optional bound constraints. Inspired by the Gauss-Newton method, DFO-LS constructs simplified linear regression models for the residuals. DFO-LS allows flexible initialization for expensive problems, whereby it can begin making progress from as few as two objective evaluations. Numerical results show DFO-LS can gain reasonable progress on some medium-scale problems with fewer objective evaluations than is needed for one gradient evaluation. DFO-LS has improved robustness to noise, allowing sample averaging, the construction of regression-based models, and multiple restart strategies together with an auto-detection mechanism. Our extensive numerical experimentation shows that restarting the solver when stagnation is detected is a cheap and effective mechanism for achieving robustness, with superior performance over both sampling and regression techniques. We also present our package Py-BOBYQA, a Python implementation of BOBYQA (Powell, 2009), which also implements robustness to noise strategies. Our numerical experiments show that Py-BOBYQA is comparable to or better than existing general DFO solvers for noisy problems. In our comparisons, we introduce a new adaptive measure of accuracy for the data profiles of noisy functions that strikes a balance between measuring the true and the noisy objective improvement.