# Pathfinder: Parallel quasi-Newton variational inference arXiv: https://arxiv.org/abs/2108.03782 · HTML: https://mlsanity.com/paper/2108.03782/ ## Key facts - Cited by 26 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 67 references that are in the map. - First posted on arXiv 9 Aug 2021. ## Builds on (most cited first, 22 of 22) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015) - [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016) - [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018) - [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013) - [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019) - [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017) - [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016) - [TensorFlow Distributions](https://mlsanity.com/paper/1711.10604/) (2017) - [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019) - [Hilbert Space Methods for Reduced-Rank Gaussian Process Regression](https://mlsanity.com/paper/1401.5508/) (2014) - [Yes, but Did It Work?: Evaluating Variational Inference](https://mlsanity.com/paper/1802.02538/) (2018) - [Hamiltonian Monte Carlo for Hierarchical Models](https://mlsanity.com/paper/1312.0906/) (2013) - [Bayesian Workflow](https://mlsanity.com/paper/2011.01808/) (2020) - [Sparsity information and regularization in the horseshoe and other shrinkage priors](https://mlsanity.com/paper/1707.01694/) (2017) - [Pareto Smoothed Importance Sampling](https://mlsanity.com/paper/1507.02646/) (2015) - [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021) - [Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming](https://mlsanity.com/paper/2004.11408/) (2020) - [Patterns of Scalable Bayesian Inference](https://mlsanity.com/paper/1602.05221/) (2016) - [Early Stopping is Nonparametric Variational Inference](https://mlsanity.com/paper/1504.01344/) (2015) - [Robust, Accurate Stochastic Optimization for Variational Inference](https://mlsanity.com/paper/2009.00666/) (2020) - [Selecting the Metric in Hamiltonian Monte Carlo](https://mlsanity.com/paper/1905.11916/) (2019) ## Built on by (most cited first, 26 of 26) - [Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Box](https://mlsanity.com/paper/2304.05527/) (2023) - [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023) - [Amortized Bayesian Multilevel Models](https://mlsanity.com/paper/2408.13230/) (2024) - [Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs](https://mlsanity.com/paper/2403.13748/) (2024) - [EigenVI: score-based variational inference with orthogonal function expansions](https://mlsanity.com/paper/2410.24054/) (2024) - [The Shrinkage-Delinkage Trade-off: An Analysis of Factorized Gaussian Approximations for Variational Inference](https://mlsanity.com/paper/2302.09163/) (2023) - [Partially factorized variational inference for high-dimensional mixed models](https://mlsanity.com/paper/2312.13148/) (2023) - [Markovian Flow Matching: Accelerating MCMC with Continuous Normalizing Flows](https://mlsanity.com/paper/2405.14392/) (2024) - [Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix](https://mlsanity.com/paper/2410.11067/) (2024) - [Riemannian Laplace Approximation with the Fisher Metric](https://mlsanity.com/paper/2311.02766/) (2023) - [Foundation Posteriors for Approximate Probabilistic Inference](https://mlsanity.com/paper/2205.09735/) (2022) - [Sample Average Approximation for Black-Box VI](https://mlsanity.com/paper/2304.06803/) (2023) - [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024) - [Interpolating between sampling and variational inference with infinite stochastic mixtures](https://mlsanity.com/paper/2110.09618/) (2021) - [Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature](https://mlsanity.com/paper/2303.05263/) (2023) - [Adaptive importance sampling for heavy-tailed distributions via α-divergence minimization](https://mlsanity.com/paper/2310.16653/) (2023) - [Amortized Bayesian Workflow](https://mlsanity.com/paper/2409.04332/) (2024) - [Posterior SBC: Simulation-Based Calibration Checking Conditional on Data](https://mlsanity.com/paper/2502.03279/) (2025) - [Flow Annealed Kalman Inversion for Gradient-Free Inference in Bayesian Inverse Problems](https://mlsanity.com/paper/2309.11490/) (2023) - [Simplifying deflation for non-convex optimization with applications in Bayesian inference and topology optimization](https://mlsanity.com/paper/2201.11926/) (2022) - [Fast inference of latent space dynamics in huge relational event networks](https://mlsanity.com/paper/2303.17460/) (2023) - [VISA: Variational Inference with Sequential Sample-Average Approximations](https://mlsanity.com/paper/2403.09429/) (2024) - [Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference](https://mlsanity.com/paper/2505.21721/) (2025) - [Large-Scale Bayesian Causal Discovery with Interventional Data](https://mlsanity.com/paper/2510.01562/) (2025) - [A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression](https://mlsanity.com/paper/2601.21217/) (2026) - [Pareto-Optimal Anytime Algorithms via Bayesian Racing](https://mlsanity.com/paper/2603.08493/) (2026) ## Abstract (arXiv) We propose Pathfinder, a variational method for approximately sampling from differentiable log densities. Starting from a random initialization, Pathfinder locates normal approximations to the target density along a quasi-Newton optimization path, with local covariance estimated using the inverse Hessian estimates produced by the optimizer. Pathfinder returns draws from the approximation with the lowest estimated Kullback-Leibler (KL) divergence to the true posterior. We evaluate Pathfinder on a wide range of posterior distributions, demonstrating that its approximate draws are better than those from automatic differentiation variational inference (ADVI) and comparable to those produced by short chains of dynamic Hamiltonian Monte Carlo (HMC), as measured by 1-Wasserstein distance. Compared to ADVI and short dynamic HMC runs, Pathfinder requires one to two orders of magnitude fewer log density and gradient evaluations, with greater reductions for more challenging posteriors. Importance resampling over multiple runs of Pathfinder improves the diversity of approximate draws, reducing 1-Wasserstein distance further and providing a measure of robustness to optimization failures on plateaus, saddle points, or in minor modes. The Monte Carlo KL divergence estimates are embarrassingly parallelizable in the core Pathfinder algorithm, as are multiple runs in the resampling version, further increasing Pathfinder's speed advantage with multiple cores.