# Loss function based second-order Jensen inequality and its application to particle variational inference

arXiv: https://arxiv.org/abs/2106.05010 · HTML: https://mlsanity.com/paper/2106.05010/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [PAC-Bayesian Theory Meets Bayesian Inference](https://mlsanity.com/paper/1605.08636/) (2016)
- [A Non-Asymptotic Analysis for Stein Variational Gradient Descent](https://mlsanity.com/paper/2006.09797/) (2020)
- [Gradient Estimators for Implicit Models](https://mlsanity.com/paper/1705.07107/) (2017)
- [Stein Variational Policy Gradient](https://mlsanity.com/paper/1704.02399/) (2017)
- [A Unified Particle-Optimization Framework for Scalable Bayesian Sampling](https://mlsanity.com/paper/1805.11659/) (2018)
- [Function Space Particle Optimization for Bayesian Neural Networks](https://mlsanity.com/paper/1902.09754/) (2019)
- [Second Order PAC-Bayesian Bounds for the Weighted Majority Vote](https://mlsanity.com/paper/2007.13532/) (2020)
- [GDPP: Learning Diverse Generations Using Determinantal Point Process](https://mlsanity.com/paper/1812.00068/) (2018)
- [Note on the bias and variance of variational inference](https://mlsanity.com/paper/1906.03708/) (2019)

## Built on by (most cited first, 3 of 3)

- [Excess risk analysis for epistemic uncertainty with application to variational inference](https://mlsanity.com/paper/2206.01606/) (2022)
- [Meta-learning to Calibrate Gaussian Processes with Deep Kernels for Regression Uncertainty Estimation](https://mlsanity.com/paper/2312.07952/) (2023)
- [A Divergence-Based Method for Weighting and Averaging Model Predictions](https://mlsanity.com/paper/2604.24172/) (2026)

## Abstract (arXiv)

Bayesian model averaging, obtained as the expectation of a likelihood function by a posterior distribution, has been widely used for prediction, evaluation of uncertainty, and model selection. Various approaches have been developed to efficiently capture the information in the posterior distribution; one such approach is the optimization of a set of models simultaneously with interaction to ensure the diversity of the individual models in the same way as ensemble learning. A representative approach is particle variational inference (PVI), which uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. However, despite its promising performance, a theoretical understanding of this repulsion and its association with the generalization ability remains unclear. In this paper, we tackle this problem in light of PAC-Bayesian analysis. First, we provide a new second-order Jensen inequality, which has the repulsion term based on the loss function. Thanks to the repulsion term, it is tighter than the standard Jensen inequality. Then, we derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models. Finally, we derive a new PVI that optimizes the generalization error bound directly. Numerical experiments demonstrate that the performance of the proposed PVI compares favorably with existing methods in the experiment.