# Minimum Excess Risk in Bayesian Learning

arXiv: https://arxiv.org/abs/2012.14868 · HTML: https://mlsanity.com/paper/2012.14868/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 30 of 35)

- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Optimal Representations for Covariate Shift](https://mlsanity.com/paper/2201.00057/) (2021)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency](https://mlsanity.com/paper/2502.04964/) (2025)
- [UMD: Unsupervised Model Detection for X2X Backdoor Attacks](https://mlsanity.com/paper/2305.18651/) (2023)
- [On Subjective Uncertainty Quantification and Calibration in Natural Language Generation](https://mlsanity.com/paper/2406.05213/) (2024)
- [Neural Bayes Estimators for Irregular Spatial Data using Graph Neural Networks](https://mlsanity.com/paper/2310.02600/) (2023)
- [Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning](https://mlsanity.com/paper/2105.04180/) (2021)
- [Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning](https://mlsanity.com/paper/2106.00252/) (2021)
- [Information-theoretic Generalization Analysis for Expected Calibration Error](https://mlsanity.com/paper/2405.15709/) (2024)
- [f-divergences and their applications in lossy compression and bounding generalization error](https://mlsanity.com/paper/2206.11042/) (2022)
- [When does a predictor know its own loss?](https://mlsanity.com/paper/2502.20375/) (2025)
- [Uncertainty Prioritized Experience Replay](https://mlsanity.com/paper/2506.09270/) (2025)
- [Information-Theoretic Analysis of Minimax Excess Risk](https://mlsanity.com/paper/2202.07537/) (2022)
- [A Bayesian Approach to (Online) Transfer Learning: Theory and Algorithms](https://mlsanity.com/paper/2109.01377/) (2021)
- [Information-theoretic Analysis of Test Data Sensitivity in Uncertainty](https://mlsanity.com/paper/2307.12456/) (2023)
- [On Uncertainty Quantification for Near-Bayes Optimal Algorithms](https://mlsanity.com/paper/2403.19381/) (2024)
- [A view on model misspecification in uncertainty quantification](https://mlsanity.com/paper/2210.16938/) (2022)
- [Lossless Transformations and Excess Risk Bounds in Statistical Inference](https://mlsanity.com/paper/2307.16735/) (2023)
- [Constructing Multiclass Classifiers using Binary Classifiers Under Log-Loss](https://mlsanity.com/paper/2102.08184/) (2021)
- [Bayesian Active Meta-Learning for Black-Box Optimization](https://mlsanity.com/paper/2110.09943/) (2021)
- [Information-Theoretic Bayes Risk Lower Bounds for Realizable Models](https://mlsanity.com/paper/2111.04579/) (2021)
- [An Information-Theoretic Analysis of Bayesian Reinforcement Learning](https://mlsanity.com/paper/2207.08735/) (2022)
- [Information-Theoretic Minimax Regret Bounds for Reinforcement Learning based on Duality](https://mlsanity.com/paper/2410.16013/) (2024)
- [An Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints](https://mlsanity.com/paper/2110.05014/) (2021)
- [Dynamic Inference](https://mlsanity.com/paper/2111.14746/) (2021)
- [Excess risk analysis for epistemic uncertainty with application to variational inference](https://mlsanity.com/paper/2206.01606/) (2022)
- [Bounds on the Excess Minimum Risk via Generalized Information Divergence Measures](https://mlsanity.com/paper/2505.24117/) (2025)
- [Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networks](https://mlsanity.com/paper/2512.12898/) (2025)

## Abstract (arXiv)

We analyze the best achievable performance of Bayesian learning under generative models by defining and upper-bounding the minimum excess risk (MER): the gap between the minimum expected loss attainable by learning from data and the minimum expected loss that could be achieved if the model realization were known. The definition of MER provides a principled way to define different notions of uncertainties in Bayesian learning, including the aleatoric uncertainty and the minimum epistemic uncertainty. Two methods for deriving upper bounds for the MER are presented. The first method, generally suitable for Bayesian learning with a parametric generative model, upper-bounds the MER by the conditional mutual information between the model parameters and the quantity being predicted given the observed data. It allows us to quantify the rate at which the MER decays to zero as more data becomes available. Under realizable models, this method also relates the MER to the richness of the generative function class, notably the VC dimension in binary classification. The second method, particularly suitable for Bayesian learning with a parametric predictive model, relates the MER to the minimum estimation error of the model parameters from data. It explicitly shows how the uncertainty in model parameter estimation translates to the MER and to the final prediction uncertainty. We also extend the definition and analysis of MER to the setting with multiple model families and the setting with nonparametric models. Along the discussions we draw some comparisons between the MER in Bayesian learning and the excess risk in frequentist learning.