Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimization
Not on the map. Why?
Abstract (arXiv)
We propose a novel algorithm that extends the methods of ball smoothing and Gaussian smoothing for noisy derivative-free optimization by accounting for the heterogeneous curvature of the objective function. The algorithm dynamically adapts the shape of the smoothing kernel to approximate the Hessian of the objective function around a local optimum. This approach significantly reduces the error in estimating the gradient from noisy evaluations through sampling. We demonstrate the efficacy of our method through numerical experiments on artificial problems. Additionally, we show improved performance when tuning NP-hard combinatorial optimization solvers compared to existing state-of-the-art heuristic derivative-free and Bayesian optimization methods.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 79 references that are in the map. First posted on arXiv 2 May 2024.
Builds on
29 of 79 references.
References · 29
- 2015Trust Region Policy Optimizationcited by 2,503
- 2021Convex optimizationcited by 2,479
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2018A Tutorial on Bayesian Optimizationcited by 734
- 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
- 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
- 2023Symbolic Discovery of Optimization Algorithmscited by 423
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
- 2018Learning a SAT Solver from Single-Bit Supervisioncited by 198
17 more
- 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
- 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
- 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
- 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
- 2021Combinatorial Optimization with Physics-Inspired Graph Neural Networkscited by 107
- 2015An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedbackcited by 105
- 2014ParamILS: An Automatic Algorithm Configuration Frameworkcited by 88
- 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
- 2015An optimal randomized incremental gradient methodcited by 32
- 2020On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNscited by 18
- 2022The Power of First-Order Smooth Optimization for Black-Box Non-Smooth Problemscited by 9
- 2016Learning Supervised PageRank with Gradient-Based and Gradient-Free Optimization Methodscited by 4
- 2018Optimizing Simulations with Noise-Tolerant Structured Explorationcited by 2
- 2019Adaptive Sampling Quasi-Newton Methods for Derivative-Free Stochastic Optimizationcited by 2
- 2022Fast Hyperparameter Tuning for Ising Machinescited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.