# Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models

arXiv: https://arxiv.org/abs/1703.04265 · HTML: https://mlsanity.com/paper/1703.04265/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2017.

## Built on by (most cited first, 30 of 65)

- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Spatio-Temporal Variational Gaussian Processes](https://mlsanity.com/paper/2111.01732/) (2021)
- [Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective](https://mlsanity.com/paper/2402.03496/) (2024)
- [Conditioning Sparse Variational Gaussian Processes for Online Decision-making](https://mlsanity.com/paper/2110.15172/) (2021)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [On the Convergence of Black-Box Variational Inference](https://mlsanity.com/paper/2305.15349/) (2023)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Dual Parameterization of Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2111.03412/) (2021)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance](https://mlsanity.com/paper/2302.11024/) (2023)
- [The Memory Perturbation Equation: Understanding Model's Sensitivity to Data](https://mlsanity.com/paper/2310.19273/) (2023)
- [Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization](https://mlsanity.com/paper/2509.03378/) (2025)
- [Bayesian Online Natural Gradient (BONG)](https://mlsanity.com/paper/2405.19681/) (2024)
- [Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees](https://mlsanity.com/paper/2111.01721/) (2021)
- [Partitioned Variational Inference: A Framework for Probabilistic Federated Learning](https://mlsanity.com/paper/2202.12275/) (2022)
- [Bayesian Continual Learning via Spiking Neural Networks](https://mlsanity.com/paper/2208.13723/) (2022)
- [Revisiting Structured Variational Autoencoders](https://mlsanity.com/paper/2305.16543/) (2023)
- [Variational Gaussian Process Diffusion Processes](https://mlsanity.com/paper/2306.02066/) (2023)
- [A practical tutorial on Variational Bayes](https://mlsanity.com/paper/2103.01327/) (2021)
- [Sparse Algorithms for Markovian Gaussian Processes](https://mlsanity.com/paper/2103.10710/) (2021)
- [Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning](https://mlsanity.com/paper/2302.09738/) (2023)
- [Bayesian Variational Federated Learning and Unlearning in Decentralized Networks](https://mlsanity.com/paper/2104.03834/) (2021)
- [Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC](https://mlsanity.com/paper/2312.05705/) (2023)
- [Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flows](https://mlsanity.com/paper/2406.17263/) (2024)
- [Function-space Parameterization of Neural Networks for Sequential Learning](https://mlsanity.com/paper/2403.10929/) (2024)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Combining Pseudo-Point and State Space Approximations for Sum-Separable Gaussian Processes](https://mlsanity.com/paper/2106.10210/) (2021)
- [Fantasizing with Dual GPs in Bayesian Optimization and Active Learning](https://mlsanity.com/paper/2211.01053/) (2022)

## Abstract (arXiv)

Variational inference is computationally challenging in models that contain both conjugate and non-conjugate terms. Methods specifically designed for conjugate models, even though computationally efficient, find it difficult to deal with non-conjugate terms. On the other hand, stochastic-gradient methods can handle the non-conjugate terms but they usually ignore the conjugate structure of the model which might result in slow convergence. In this paper, we propose a new algorithm called Conjugate-computation Variational Inference (CVI) which brings the best of the two worlds together -- it uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest. We derive this algorithm by using a stochastic mirror-descent method in the mean-parameter space, and then expressing each gradient step as a variational inference in a conjugate model. We demonstrate our algorithm's applicability to a large class of models and establish its convergence. Our experimental results show that our method converges much faster than the methods that ignore the conjugate structure of the model.