# Markovian Score Climbing: Variational Inference with KL(p||q)

arXiv: https://arxiv.org/abs/2003.10374 · HTML: https://mlsanity.com/paper/2003.10374/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 30 of 46)

- [Training Chain-of-Thought via Latent-Variable Inference](https://mlsanity.com/paper/2312.02179/) (2023)
- [A Variational Perspective on Generative Flow Networks](https://mlsanity.com/paper/2210.07992/) (2022)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [Local-Global MCMC kernels: the best of both worlds](https://mlsanity.com/paper/2111.02702/) (2021)
- [On the Convergence of Black-Box Variational Inference](https://mlsanity.com/paper/2305.15349/) (2023)
- [Sliced Wasserstein Variational Inference](https://mlsanity.com/paper/2207.13177/) (2022)
- [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023)
- [Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference](https://mlsanity.com/paper/2407.17033/) (2024)
- [Conditioning Diffusions Using Malliavin Calculus](https://mlsanity.com/paper/2504.03461/) (2025)
- [BR-SNIS: Bias Reduced Self-Normalized Importance Sampling](https://mlsanity.com/paper/2207.06364/) (2022)
- [Nesting Particle Filters for Experimental Design in Dynamical Systems](https://mlsanity.com/paper/2402.07868/) (2024)
- [Variational Combinatorial Sequential Monte Carlo Methods for Bayesian Phylogenetic Inference](https://mlsanity.com/paper/2106.00075/) (2021)
- [Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs](https://mlsanity.com/paper/2403.13748/) (2024)
- [Empirical Evaluation of Biased Methods for Alpha Divergence Minimization](https://mlsanity.com/paper/2105.06587/) (2021)
- [Markovian Flow Matching: Accelerating MCMC with Continuous Normalizing Flows](https://mlsanity.com/paper/2405.14392/) (2024)
- [Transport Score Climbing: Variational Inference Using Forward KL and Adaptive Neural Transport](https://mlsanity.com/paper/2202.01841/) (2022)
- [Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows](https://mlsanity.com/paper/2310.20090/) (2023)
- [Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference](https://mlsanity.com/paper/2403.10610/) (2024)
- [Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients](https://mlsanity.com/paper/2206.06295/) (2022)
- [Variational Inference with Locally Enhanced Bounds for Hierarchical Models](https://mlsanity.com/paper/2203.04432/) (2022)
- [State and parameter learning with PaRIS particle Gibbs](https://mlsanity.com/paper/2301.00900/) (2023)
- [Balanced Training of Energy-Based Models with Adaptive Flow Sampling](https://mlsanity.com/paper/2306.00684/) (2023)
- [Probabilistic Programming with Programmable Variational Inference](https://mlsanity.com/paper/2406.15742/) (2024)
- [Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling](https://mlsanity.com/paper/2408.06710/) (2024)
- [Inclusive KL Gradient Flows: Otto-Wasserstein, Fisher-Rao-Gaussian, and Local-Estimator Dynamics](https://mlsanity.com/paper/2411.00214/) (2024)
- [End-To-End Learning of Gaussian Mixture Priors for Diffusion Sampler](https://mlsanity.com/paper/2503.00524/) (2025)
- [Advancing Semi-Supervised Task Oriented Dialog Systems by JSA Learning of Discrete Latent Variable Models](https://mlsanity.com/paper/2207.12235/) (2022)
- [Recursive Nested Filtering for Efficient Amortized Bayesian Experimental Design](https://mlsanity.com/paper/2409.05354/) (2024)
- [Preference learning made easy: Everything should be understood through win rate](https://mlsanity.com/paper/2502.10505/) (2025)
- [Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry](https://mlsanity.com/paper/2511.01064/) (2025)

## Abstract (arXiv)

Modern variational inference (VI) uses stochastic gradients to avoid intractable expectations, enabling large-scale probabilistic inference in complex models. VI posits a family of approximating distributions q and then finds the member of that family that is closest to the exact posterior p. Traditionally, VI algorithms minimize the "exclusive Kullback-Leibler (KL)" KL(q || p), often for computational convenience. Recent research, however, has also focused on the "inclusive KL" KL(p || q), which has good statistical properties that makes it more appropriate for certain inference problems. This paper develops a simple algorithm for reliably minimizing the inclusive KL using stochastic gradients with vanishing bias. This method, which we call Markovian score climbing (MSC), converges to a local optimum of the inclusive KL. It does not suffer from the systematic errors inherent in existing methods, such as Reweighted Wake-Sleep and Neural Adaptive Sequential Monte Carlo, which lead to bias in their final estimates. We illustrate convergence on a toy model and demonstrate the utility of MSC on Bayesian probit regression for classification as well as a stochastic volatility model for financial data.