# Boosting Variational Inference With Locally Adaptive Step-Sizes

arXiv: https://arxiv.org/abs/2105.09240 · HTML: https://mlsanity.com/paper/2105.09240/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 17 of 17)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [SOM-VAE: Interpretable Discrete Representation Learning on Time Series](https://mlsanity.com/paper/1806.02199/) (2018)
- [Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations](https://mlsanity.com/paper/1906.02914/) (2019)
- [Variational Boosting: Iteratively Refining Posterior Approximations](https://mlsanity.com/paper/1611.06585/) (2016)
- [Edward: A library for probabilistic modeling, inference, and criticism](https://mlsanity.com/paper/1610.09787/) (2016)
- [Boosting Variational Inference](https://mlsanity.com/paper/1611.05559/) (2016)
- [Boosting Black Box Variational Inference](https://mlsanity.com/paper/1806.02185/) (2018)
- [Variational Particle Approximations](https://mlsanity.com/paper/1402.5715/) (2014)
- [A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe](https://mlsanity.com/paper/1702.06457/) (2017)
- [Linearly Convergent Frank-Wolfe with Backtracking Line-Search](https://mlsanity.com/paper/1806.05123/) (2018)
- [Universal Boosting Variational Inference](https://mlsanity.com/paper/1906.01235/) (2019)
- [Boosting Variational Inference: an Optimization Perspective](https://mlsanity.com/paper/1708.01733/) (2017)
- [Boosted Density Estimation Remastered](https://mlsanity.com/paper/1803.08178/) (2018)
- [A Stochastic Line Search Method with Convergence Rate Analysis](https://mlsanity.com/paper/1807.07994/) (2018)
- [Variational Inference via Transformations on Distributions](https://mlsanity.com/paper/1707.02510/) (2017)

## Abstract (arXiv)

Variational Inference makes a trade-off between the capacity of the variational family and the tractability of finding an approximate posterior distribution. Instead, Boosting Variational Inference allows practitioners to obtain increasingly good posterior approximations by spending more compute. The main obstacle to widespread adoption of Boosting Variational Inference is the amount of resources necessary to improve over a strong Variational Inference baseline. In our work, we trace this limitation back to the global curvature of the KL-divergence. We characterize how the global curvature impacts time and memory consumption, address the problem with the notion of local curvature, and provide a novel approximate backtracking algorithm for estimating local curvature. We give new theoretical convergence rates for our algorithms and provide experimental validation on synthetic and real-world datasets.