# Variational approximations using Fisher divergence

arXiv: https://arxiv.org/abs/1905.05284 · HTML: https://mlsanity.com/paper/1905.05284/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2019.

## Built on by (most cited first, 13 of 13)

- [Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation](https://mlsanity.com/paper/2404.04057/) (2024)
- [Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests](https://mlsanity.com/paper/2107.06929/) (2021)
- [Semi-Implicit Variational Inference via Score Matching](https://mlsanity.com/paper/2308.10014/) (2023)
- [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023)
- [Provable Separations between Memorization and Generalization in Diffusion Models](https://mlsanity.com/paper/2511.03202/) (2025)
- [Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation](https://mlsanity.com/paper/2406.01561/) (2024)
- [EigenVI: score-based variational inference with orthogonal function expansions](https://mlsanity.com/paper/2410.24054/) (2024)
- [Batch, match, and patch: low-rank approximations for score-based variational inference](https://mlsanity.com/paper/2410.22292/) (2024)
- [Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network](https://mlsanity.com/paper/2304.12770/) (2023)
- [Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities](https://mlsanity.com/paper/2107.05901/) (2021)
- [Multiple Instance Learning with Auxiliary Task Weighting for Multiple Myeloma Classification](https://mlsanity.com/paper/2107.07805/) (2021)
- [How to Approximate Inference with Subtractive Mixture Models](https://mlsanity.com/paper/2604.16714/) (2026)
- [Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence](https://mlsanity.com/paper/2606.19876/) (2026)

## Abstract (arXiv)

Modern applications of Bayesian inference involve models that are sufficiently complex that the corresponding posterior distributions are intractable and must be approximated. The most common approximation is based on Markov chain Monte Carlo, but these can be expensive when the data set is large and/or the model is complex, so more efficient variational approximations have recently received considerable attention. The traditional variational methods, that seek to minimize the Kullback--Leibler divergence between the posterior and a relatively simple parametric family, provide accurate and efficient estimation of the posterior mean, but often does not capture other moments, and have limitations in terms of the models to which they can be applied. Here we propose the construction of variational approximations based on minimizing the Fisher divergence, and develop an efficient computational algorithm that can be applied to a wide range of models without conjugacy or potentially unrealistic mean-field assumptions. We demonstrate the superior performance of the proposed method for the benchmark case of logistic regression.