# Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning

arXiv: https://arxiv.org/abs/2106.00252 · HTML: https://mlsanity.com/paper/2106.00252/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [Bayesian Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.03836/) (2018)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [Reasoning About Generalization via Conditional Mutual Information](https://mlsanity.com/paper/2001.09122/) (2020)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates](https://mlsanity.com/paper/1911.02151/) (2019)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [Meta-Learning Probabilistic Inference For Prediction](https://mlsanity.com/paper/1805.09921/) (2018)
- [PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees](https://mlsanity.com/paper/2002.05551/) (2020)
- [Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory](https://mlsanity.com/paper/1711.01244/) (2017)
- [A PAC-Bayesian bound for Lifelong Learning](https://mlsanity.com/paper/1311.2838/) (2013)
- [The Case for Bayesian Deep Learning](https://mlsanity.com/paper/2001.10995/) (2020)
- [CCMI : Classifier based Conditional Mutual Information Estimation](https://mlsanity.com/paper/1906.01824/) (2019)
- [Minimum Excess Risk in Bayesian Learning](https://mlsanity.com/paper/2012.14868/) (2020)
- [Information-Theoretic Generalization Bounds for Meta-Learning and Applications](https://mlsanity.com/paper/2005.04372/) (2020)
- [Riemannian Stein Variational Gradient Descent for Bayesian Inference](https://mlsanity.com/paper/1711.11216/) (2017)
- [Uncertainty in Model-Agnostic Meta-Learning using Variational Inference](https://mlsanity.com/paper/1907.11864/) (2019)
- [An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning](https://mlsanity.com/paper/2101.08390/) (2021)

## Built on by (most cited first, 10 of 10)

- [Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction](https://mlsanity.com/paper/2210.03067/) (2022)
- [Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practice](https://mlsanity.com/paper/2211.07206/) (2022)
- [Robust PACm: Training Ensemble Models Under Misspecification and Outliers](https://mlsanity.com/paper/2203.01859/) (2022)
- [Information-theoretic Analysis of Test Data Sensitivity in Uncertainty](https://mlsanity.com/paper/2307.12456/) (2023)
- [Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization](https://mlsanity.com/paper/2108.00785/) (2021)
- [Predicting Multi-Antenna Frequency-Selective Channels via Meta-Learned Linear Filters based on Long-Short Term Channel Decomposition](https://mlsanity.com/paper/2203.12715/) (2022)
- [Bayesian Active Meta-Learning for Black-Box Optimization](https://mlsanity.com/paper/2110.09943/) (2021)
- [Optimizing In-Context Learning for Efficient Full Conformal Prediction](https://mlsanity.com/paper/2509.01840/) (2025)
- [Excess risk analysis for epistemic uncertainty with application to variational inference](https://mlsanity.com/paper/2206.01606/) (2022)
- [Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI](https://mlsanity.com/paper/2511.18517/) (2025)

## Abstract (arXiv)

The overall predictive uncertainty of a trained predictor can be decomposed into separate contributions due to epistemic and aleatoric uncertainty. Under a Bayesian formulation, assuming a well-specified model, the two contributions can be exactly expressed (for the log-loss) or bounded (for more general losses) in terms of information-theoretic quantities (Xu and Raginsky, 2020). This paper addresses the study of epistemic uncertainty within an information-theoretic framework in the broader setting of Bayesian meta-learning. A general hierarchical Bayesian model is assumed in which hyperparameters determine the per-task priors of the model parameters. Exact characterizations (for the log-loss) and bounds (for more general losses) are derived for the epistemic uncertainty -quantified by the minimum excess meta-risk (MEMR)- of optimal meta-learning rules. This characterization is leveraged to bring insights into the dependence of the epistemic uncertainty on the number of tasks and on the amount of per-task training data. Experiments are presented that use the proposed information-theoretic bounds, evaluated via neural mutual information estimators, to compare the performance of conventional learning and meta-learning as the number of meta-learning tasks increases.