# Stochastic Variational Optimization

arXiv: https://arxiv.org/abs/1809.04855 · HTML: https://mlsanity.com/paper/1809.04855/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2018.

## Built on by (most cited first, 5 of 5)

- [Variational Search Distributions](https://mlsanity.com/paper/2409.06142/) (2024)
- [Training Hybrid Classical-Quantum Classifiers via Stochastic Variational Optimization](https://mlsanity.com/paper/2201.08629/) (2022)
- [Multi-fidelity Constrained Optimization for Stochastic Black Box Simulators](https://mlsanity.com/paper/2311.15137/) (2023)
- [From concrete mixture to structural design -- a holistic optimization procedure in the presence of uncertainties](https://mlsanity.com/paper/2312.03607/) (2023)
- [Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling](https://mlsanity.com/paper/2608.23048/) (2026)

## Abstract (arXiv)

Variational Optimization forms a differentiable upper bound on an objective. We show that approaches such as Natural Evolution Strategies and Gaussian Perturbation, are special cases of Variational Optimization in which the expectations are approximated by Gaussian sampling. These approaches are of particular interest because they are parallelizable. We calculate the approximate bias and variance of the corresponding gradient estimators and demonstrate that using antithetic sampling or a baseline is crucial to mitigate their problems. We contrast these methods with an alternative parallelizable method, namely Directional Derivatives. We conclude that, for differentiable objectives, using Directional Derivatives is preferable to using Variational Optimization to perform parallel Stochastic Gradient Descent.