# Efficient Semi-Implicit Variational Inference

arXiv: https://arxiv.org/abs/2101.06070 · HTML: https://mlsanity.com/paper/2101.06070/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 15 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator](https://mlsanity.com/paper/1807.01695/) (2018)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Learning in Implicit Generative Models](https://mlsanity.com/paper/1610.03483/) (2016)
- [Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functions](https://mlsanity.com/paper/1411.3803/) (2014)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Auxiliary Deep Generative Models](https://mlsanity.com/paper/1602.05473/) (2016)
- [Variance Reduction for Faster Non-Convex Optimization](https://mlsanity.com/paper/1603.05643/) (2016)
- [Gradient Estimators for Implicit Models](https://mlsanity.com/paper/1705.07107/) (2017)
- [Semi-Implicit Variational Inference](https://mlsanity.com/paper/1805.11183/) (2018)
- [Variants of RMSProp and Adagrad with Logarithmic Regret Bounds](https://mlsanity.com/paper/1706.05507/) (2017)
- [Accelerating Stochastic Composition Optimization](https://mlsanity.com/paper/1607.07329/) (2016)
- [Variational Inference using Implicit Distributions](https://mlsanity.com/paper/1702.08235/) (2017)
- [Unbiased Implicit Variational Inference](https://mlsanity.com/paper/1808.02078/) (2018)
- [The Variational Gaussian Process](https://mlsanity.com/paper/1511.06499/) (2015)
- [Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes](https://mlsanity.com/paper/1506.04088/) (2015)
- [Variational Gaussian Copula Inference](https://mlsanity.com/paper/1506.05860/) (2015)
- [Multi-View Reinforcement Learning](https://mlsanity.com/paper/1910.08285/) (2019)
- [Compositional ADAM: An Adaptive Compositional Solver](https://mlsanity.com/paper/2002.03755/) (2020)
- [Stochastically Controlled Stochastic Gradient for the Convex and Non-convex Composition problem](https://mlsanity.com/paper/1809.02505/) (2018)
- [A Distributed Newton Method for Large Scale Consensus Optimization](https://mlsanity.com/paper/1606.06593/) (2016)
- [Derivative-Free & Order-Robust Optimisation](https://mlsanity.com/paper/1910.04034/) (2019)

## Built on by (most cited first, 9 of 9)

- [Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation](https://mlsanity.com/paper/2404.04057/) (2024)
- [Semi-Implicit Variational Inference via Score Matching](https://mlsanity.com/paper/2308.10014/) (2023)
- [Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration](https://mlsanity.com/paper/2310.17153/) (2023)
- [Kernel Semi-Implicit Variational Inference](https://mlsanity.com/paper/2405.18997/) (2024)
- [VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognition](https://mlsanity.com/paper/2204.00130/) (2022)
- [A Kernel Approach for Semi-implicit Variational Inference](https://mlsanity.com/paper/2601.12023/) (2026)
- [Bayesian Neural Network Inference via Implicit Models and the Posterior Predictive Distribution](https://mlsanity.com/paper/2209.02188/) (2022)
- [Continuous Semi-Implicit Models](https://mlsanity.com/paper/2506.06778/) (2025)
- [Implicit Variational Rejection Sampling](https://mlsanity.com/paper/2606.14235/) (2026)

## Abstract (arXiv)

In this paper, we propose CI-VI an efficient and scalable solver for semi-implicit variational inference (SIVI). Our method, first, maps SIVI's evidence lower bound (ELBO) to a form involving a nonlinear functional nesting of expected values and then develops a rigorous optimiser capable of correctly handling bias inherent to nonlinear nested expectations using an extrapolation-smoothing mechanism coupled with gradient sketching. Our theoretical results demonstrate convergence to a stationary point of the ELBO in general non-convex settings typically arising when using deep network models and an order of O(t^-frac45) gradient-bias-vanishing rate. We believe these results generalise beyond the specific nesting arising from SIVI to other forms. Finally, in a set of experiments, we demonstrate the effectiveness of our algorithm in approximating complex posteriors on various data-sets including those from natural language processing.