Likelihood approximations via Gaussian approximate inference
Not on the map. Why?
Abstract (arXiv)
Non-Gaussian likelihoods are essential for modelling complex real-world observations but pose significant computational challenges in learning and inference. Even with Gaussian priors, non-Gaussian likelihoods often lead to analytically intractable posteriors, necessitating approximation methods. To this end, we propose efficient schemes to approximate the effects of non-Gaussian likelihoods by Gaussian densities based on variational inference and moment matching in transformed bases. These enable efficient inference strategies originally designed for models with a Gaussian likelihood to be deployed. Our empirical results demonstrate that the proposed matching strategies attain good approximation quality for binary and multiclass classification in large-scale point-estimate and distributional inferential settings. In challenging streaming problems, the proposed methods outperform all existing likelihood approximations and approximate inference methods in the exact models. As a by-product, we show that the proposed approximate log-likelihoods are a superior alternative to least-squares on raw labels for neural network classification.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 28 Oct 2024.
Builds on
18 of 33 references.
References · 18
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2016Variational Inference: A Review for Statisticianscited by 1,466
- 2017Deep Bayesian Active Learning with Image Datacited by 812
- 2015Deep Kernel Learningcited by 440
- 2017Variational Continual Learningcited by 335
- 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
- 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
- 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
- 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
- 2018Dirichlet-based Gaussian Processes for Large-scale Calibrated Classificationcited by 50
- 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
6 more
- 2017Streaming Sparse Gaussian Process Approximationscited by 41
- 2020Variational Auto-Regressive Gaussian Processes for Continual Learningcited by 23
- 2023Kalman Filter for Online Classification of Non-Stationary Datacited by 13
- 2023Gaussian Process Probes (GPP) for Uncertainty-Aware Probingcited by 11
- 2021Laplace Matching for fast Approximate Inference in Latent Gaussian Modelscited by 3
- 2023Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Modelscited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.