# Projective Integral Updates for High-Dimensional Variational Inference

arXiv: https://arxiv.org/abs/2301.08374 · HTML: https://mlsanity.com/paper/2301.08374/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [A Tensorized Transformer for Language Modeling](https://mlsanity.com/paper/1906.09777/) (2019)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee](https://mlsanity.com/paper/2011.07439/) (2020)
- [Variational Tempering](https://mlsanity.com/paper/1411.1810/) (2014)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Sigma-Point Filtering and Smoothing Based Parameter Estimation in Nonlinear Dynamic Systems](https://mlsanity.com/paper/1504.06173/) (2015)
- [Generalizing Information to the Evolution of Rational Belief](https://mlsanity.com/paper/1911.09559/) (2019)

## Built on by (most cited first, 1 of 1)

- [Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscape](https://mlsanity.com/paper/2411.16914/) (2024)

## Abstract (arXiv)

Variational inference is an approximation framework for Bayesian inference that seeks to improve quantified uncertainty in predictions by optimizing a simplified distribution over parameters to stand in for the full posterior. Capturing model variations that remain consistent with training data enables more robust predictions by reducing parameter sensitivity. This work introduces a fixed-point optimization for variational inference that is applicable when every feasible log density can be expressed as a linear combination of functions from a given basis. In such cases, the optimizer becomes a fixed-point of projective integral updates. When the basis spans univariate quadratics in each parameter, feasible densities are Gaussian and the projective integral updates yield quasi-Newton variational Bayes (QNVB). Other bases and updates are also possible. As these updates require high-dimensional integration, this work first proposes an efficient quasirandom quadrature sequence for mean-field distributions. Each iterate of the sequence contains two evaluation points that combine to correctly integrate all univariate quadratics and, if the mean-field factors are symmetric, all univariate cubics. More importantly, averaging results over short subsequences achieves periodic exactness on a much larger space of multivariate quadratics. The corresponding variational updates require 4 loss evaluations with standard (not second-order) backpropagation to eliminate error terms from over half of all multivariate quadratic basis functions. This integration technique is motivated by first proposing stochastic blocked mean-field quadratures, which may be useful in other contexts. A PyTorch implementation of QNVB allows for better control over model uncertainty during training than competing methods. Experiments demonstrate superior generalizability for multiple learning problems and architectures.