# Amortized variance reduction for doubly stochastic objectives

arXiv: https://arxiv.org/abs/2003.04125 · HTML: https://mlsanity.com/paper/2003.04125/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 3 of 3)

- [Variance reduction of diffusion model's gradients with Taylor approximation-based control variate](https://mlsanity.com/paper/2408.12270/) (2024)
- [Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference](https://mlsanity.com/paper/2505.21721/) (2025)
- [Joint control variate for faster black-box variational inference](https://mlsanity.com/paper/2210.07290/) (2022)

## Abstract (arXiv)

Approximate inference in complex probabilistic models such as deep Gaussian processes requires the optimisation of doubly stochastic objective functions. These objectives incorporate randomness both from mini-batch subsampling of the data and from Monte Carlo estimation of expectations. If the gradient variance is high, the stochastic optimisation problem becomes difficult with a slow rate of convergence. Control variates can be used to reduce the variance, but past approaches do not take into account how mini-batch stochasticity affects sampling stochasticity, resulting in sub-optimal variance reduction. We propose a new approach in which we use a recognition network to cheaply approximate the optimal control variate for each mini-batch, with no additional model gradient computations. We illustrate the properties of this proposal and test its performance on logistic regression and deep Gaussian processes.