# Information-Theoretic Generalization Bounds for Meta-Learning and Applications

arXiv: https://arxiv.org/abs/2005.04372 · HTML: https://mlsanity.com/paper/2005.04372/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2020.

## Built on by (most cited first, 30 of 32)

- [Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2109.14595/) (2021)
- [Out-of-Domain Robustness via Targeted Augmentations](https://mlsanity.com/paper/2302.11861/) (2023)
- [Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channels](https://mlsanity.com/paper/2102.02976/) (2021)
- [Information-Theoretic Analysis of Unsupervised Domain Adaptation](https://mlsanity.com/paper/2210.00706/) (2022)
- [Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?](https://mlsanity.com/paper/2203.03059/) (2022)
- [Understanding Benign Overfitting in Gradient-Based Meta Learning](https://mlsanity.com/paper/2206.13482/) (2022)
- [Evaluated CMI Bounds for Meta Learning: Tightness and Expressiveness](https://mlsanity.com/paper/2210.06511/) (2022)
- [Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practice](https://mlsanity.com/paper/2211.07206/) (2022)
- [An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning](https://mlsanity.com/paper/2101.08390/) (2021)
- [Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning](https://mlsanity.com/paper/2106.00252/) (2021)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [Provable Generalization of Overparameterized Meta-learning Trained with SGD](https://mlsanity.com/paper/2206.09136/) (2022)
- [Fast Power Control Adaptation via Meta-Learning for Random Edge Graph Neural Networks](https://mlsanity.com/paper/2105.00459/) (2021)
- [Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes](https://mlsanity.com/paper/2302.11709/) (2023)
- [Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation](https://mlsanity.com/paper/2304.02064/) (2023)
- [Fast On-Device Adaptation for Spiking Neural Networks via Online-Within-Online Meta-Learning](https://mlsanity.com/paper/2103.03901/) (2021)
- [On the Algorithmic Stability and Generalization of Adaptive Optimization Methods](https://mlsanity.com/paper/2211.03970/) (2022)
- [On The Statistical Limits of Self-Improving Agents](https://mlsanity.com/paper/2510.04399/) (2025)
- [Predicting Flat-Fading Channels via Meta-Learned Closed-Form Linear Filters and Equilibrium Propagation](https://mlsanity.com/paper/2110.00414/) (2021)
- [Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problem](https://mlsanity.com/paper/2305.00876/) (2023)
- [Learning to Learn Weight Generation via Local Consistency Diffusion](https://mlsanity.com/paper/2502.01117/) (2025)
- [A General framework for PAC-Bayes Bounds for Meta-Learning](https://mlsanity.com/paper/2206.05454/) (2022)
- [How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis](https://mlsanity.com/paper/2406.09745/) (2024)
- [Pre-Training and Personalized Fine-Tuning via Over-the-Air Federated Meta-Learning: Convergence-Generalization Trade-Offs](https://mlsanity.com/paper/2406.11569/) (2024)
- [Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective](https://mlsanity.com/paper/2506.07861/) (2025)
- [On the Conditioning Consistency Gap in Conditional Neural Processes](https://mlsanity.com/paper/2604.19312/) (2026)
- [On the Generalization Error of Meta Learning for the Gibbs Algorithm](https://mlsanity.com/paper/2304.14332/) (2023)
- [Lossless Transformations and Excess Risk Bounds in Statistical Inference](https://mlsanity.com/paper/2307.16735/) (2023)
- [Information-Theoretic Generalization Bounds for Transductive Learning and its Applications](https://mlsanity.com/paper/2311.04561/) (2023)
- [Unsupervised Meta-Learning via Dynamic Head and Heterogeneous Task Construction for Few-Shot Classification](https://mlsanity.com/paper/2410.02267/) (2024)

## Abstract (arXiv)

Meta-learning, or "learning to learn", refers to techniques that infer an inductive bias from data corresponding to multiple related tasks with the goal of improving the sample efficiency for new, previously unobserved, tasks. A key performance measure for meta-learning is the meta-generalization gap, that is, the difference between the average loss measured on the meta-training data and on a new, randomly selected task. This paper presents novel information-theoretic upper bounds on the meta-generalization gap. Two broad classes of meta-learning algorithms are considered that uses either separate within-task training and test sets, like MAML, or joint within-task training and test sets, like Reptile. Extending the existing work for conventional learning, an upper bound on the meta-generalization gap is derived for the former class that depends on the mutual information (MI) between the output of the meta-learning algorithm and its input meta-training data. For the latter, the derived bound includes an additional MI between the output of the per-task learning procedure and corresponding data set to capture within-task uncertainty. Tighter bounds are then developed, under given technical conditions, for the two classes via novel Individual Task MI (ITMI) bounds. Applications of the derived bounds are finally discussed, including a broad class of noisy iterative algorithms for meta-learning.