# Generalizing Gaussian Smoothing for Random Search

arXiv: https://arxiv.org/abs/2211.14721 · HTML: https://mlsanity.com/paper/2211.14721/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 27 Nov 2022.

## Builds on (most cited first, 12 of 12)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [ProMP: Proximal Meta-Policy Search](https://mlsanity.com/paper/1810.06784/) (2018)
- [Structured Evolution with Compact Architectures for Scalable Policy Optimization](https://mlsanity.com/paper/1804.02395/) (2018)
- [Stochastic Gradient Descent with Biased but Consistent Gradient Estimators](https://mlsanity.com/paper/1807.11880/) (2018)
- [Learning to Guide Random Search](https://mlsanity.com/paper/2004.12214/) (2020)
- [Modeling and Optimization Trade-off in Meta-learning](https://mlsanity.com/paper/2010.12916/) (2020)
- [(Bandit) Convex Optimization with Biased Noisy Gradient Oracles](https://mlsanity.com/paper/1609.07087/) (2016)
- [Online Variance Reduction for Stochastic Optimization](https://mlsanity.com/paper/1802.04715/) (2018)
- [Randomized Derivative-Free Optimization of Noisy Convex Functions](https://mlsanity.com/paper/1507.03332/) (2015)

## Built on by (most cited first, 15 of 15)

- [Continuation Path Learning for Homotopy Optimization](https://mlsanity.com/paper/2307.12551/) (2023)
- [Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies](https://mlsanity.com/paper/2304.12180/) (2023)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [Global Optimization with A Power-Transformed Objective and Gaussian Smoothing](https://mlsanity.com/paper/2412.05204/) (2024)
- [Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs](https://mlsanity.com/paper/2510.00419/) (2025)
- [Sharpness-Aware Black-Box Optimization](https://mlsanity.com/paper/2410.12457/) (2024)
- [Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations](https://mlsanity.com/paper/2510.18228/) (2025)
- [Stochastic Two Points Method for Deep Model Zeroth-order Optimization](https://mlsanity.com/paper/2402.01621/) (2024)
- [Power Homotopy for Zeroth-Order Non-Convex Optimizations](https://mlsanity.com/paper/2511.13592/) (2025)
- [Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective](https://mlsanity.com/paper/2605.03373/) (2026)
- [A Historical Trajectory Assisted Optimization Method for Zeroth-Order Federated Learning](https://mlsanity.com/paper/2409.15955/) (2024)
- [Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space](https://mlsanity.com/paper/2509.25876/) (2025)
- [Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization](https://mlsanity.com/paper/2605.14373/) (2026)
- [Nonparametric Learning and Earning with One-Point Feedback under Nonstationarity](https://mlsanity.com/paper/2605.21263/) (2026)
- [Probabilistic Smoothing with Ratio-Monotone Transforms for Global Optimization](https://mlsanity.com/paper/2605.27316/) (2026)

## Abstract (arXiv)

Gaussian smoothing (GS) is a derivative-free optimization (DFO) algorithm that estimates the gradient of an objective using perturbations of the current parameters sampled from a standard normal distribution. We generalize it to sampling perturbations from a larger family of distributions. Based on an analysis of DFO for non-convex functions, we propose to choose a distribution for perturbations that minimizes the mean squared error (MSE) of the gradient estimate. We derive three such distributions with provably smaller MSE than Gaussian smoothing. We conduct evaluations of the three sampling distributions on linear regression, reinforcement learning, and DFO benchmarks in order to validate our claims. Our proposal improves on GS with the same computational complexity, and are usually competitive with and often outperform Guided ES and Orthogonal ES, two computationally more expensive algorithms that adapt the covariance matrix of normally distributed perturbations.