mlsanityData to 7 Oct 2026

On the properties of variational approximations of Gibbs posteriors

arXiv 1506.04091 ↗First posted 12 Jun 2015stat.ML math.ST stat.THcited by 121

Not on the map. Why?

Abstract (arXiv)

The PAC-Bayesian approach is a powerful set of techniques to derive non- asymptotic risk bounds for random estimators. The corresponding optimal distribution of estimators, usually called the Gibbs posterior, is unfortunately intractable. One may sample from it using Markov chain Monte Carlo, but this is often too slow for big datasets. We consider instead variational approximations of the Gibbs posterior, which are fast to compute. We undertake a general study of the properties of such approximations. Our main finding is that such a variational approximation has often the same rate of convergence as the original PAC-Bayesian procedure it approximates. We specialise our results to several learning tasks (classification, ranking, matrix completion),discuss how to implement a variational approximation in each case, and illustrate the good properties of said approximation on real datasets.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 21; 2023: 28; 2024: 23; 2025: 19; 2026: 11.

121 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2022Variational inference via Wasserstein gradient flowscited by 84
  3. 2021Subgroup Generalization and Fairness of Graph Neural Networkscited by 69
  4. 2023Uncertainty Estimation by Fisher Information-based Evidential Deep Learningcited by 58
  5. 2021Robust Generalised Bayesian Inference for Intractable Likelihoodscited by 53
  6. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  7. 2022Enhancing Adversarial Training with Second-Order Statistics of Weightscited by 42
  8. 2023A unified recipe for deriving (time-uniform) PAC-Bayes boundscited by 25
  9. 2022PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingalescited by 23
  10. 2022Online PAC-Bayes Learningcited by 22
  11. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  12. 2021Learning PAC-Bayes Priors for Probabilistic Neural Networkscited by 21
38 more
  1. 2023Coherent Soft Imitation Learningcited by 21
  2. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  3. 2021Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Boundcited by 16
  4. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  5. 2021Information Complexity and Generalization Boundscited by 14
  6. 2021Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithmcited by 14
  7. 2021Learning Augmentation Distributions using Transformed Risk Minimizationcited by 13
  8. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  9. 2022Empirical Risk Minimization with Relative Entropy Regularizationcited by 13
  10. 2023Tighter PAC-Bayes Bounds Through Coin-Bettingcited by 13
  11. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  12. 2021PAC-Bayes Bounds for Meta-learning with Data-Dependent Priorcited by 11
  13. 2022Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practicecited by 11
  14. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  15. 2024Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learningcited by 11
  16. 2021Variational Inference in high-dimensional linear regressioncited by 10
  17. 2021Wide stochastic networks: Gaussian limit and PAC-Bayesian trainingcited by 10
  18. 2022Change of measure through the Legendre transformcited by 10
  19. 2022Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processescited by 10
  20. 2023Flat Seeking Bayesian Neural Networkscited by 10
  21. 2023On the Validation of Gibbs Algorithms: Training Datasets, Test Datasets and their Aggregationcited by 10
  22. 2022Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspectivecited by 9
  23. 2024PAC-Bayes-Chernoff bounds for unbounded lossescited by 9
  24. 2023Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayescited by 8
  25. 2023A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regressioncited by 8
  26. 2021Conditionally Gaussian PAC-Bayescited by 7
  27. 2022Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selectioncited by 7
  28. 2024Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measurescited by 7
  29. 2024Misclassification bounds for PAC-Bayesian sparse deep learningcited by 7
  30. 2021PAC-Bayesian theory for stochastic LTI systemscited by 6
  31. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  32. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
  33. 2024A sparse PAC-Bayesian approach for high-dimensional quantile predictioncited by 6
  34. 2024Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approachcited by 5
  35. 2024The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithmscited by 5
  36. 2025Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Designcited by 5
  37. 2022PAC-Bayesian Learning of Optimization Algorithmscited by 4
  38. 2023Semiparametric inference using fractional posteriorscited by 4

50 most cited of 121 All 121 (JSON)