# An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning

arXiv: https://arxiv.org/abs/2101.08390 · HTML: https://mlsanity.com/paper/2101.08390/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Reasoning About Generalization via Conditional Mutual Information](https://mlsanity.com/paper/2001.09122/) (2020)
- [Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates](https://mlsanity.com/paper/1911.02151/) (2019)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees](https://mlsanity.com/paper/2002.05551/) (2020)
- [Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory](https://mlsanity.com/paper/1711.01244/) (2017)
- [A PAC-Bayesian bound for Lifelong Learning](https://mlsanity.com/paper/1311.2838/) (2013)
- [Learning-to-Learn Stochastic Gradient Descent with Biased Regularization](https://mlsanity.com/paper/1903.10399/) (2019)
- [Generalization Bounds via Information Density and Conditional Information Density](https://mlsanity.com/paper/2005.08044/) (2020)
- [The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning](https://mlsanity.com/paper/2008.10857/) (2020)
- [Information-Theoretic Generalization Bounds for Meta-Learning and Applications](https://mlsanity.com/paper/2005.04372/) (2020)
- [Generalization Bounds for Domain Adaptation](https://mlsanity.com/paper/1304.1574/) (2013)
- [Information-theoretic analysis for transfer learning](https://mlsanity.com/paper/2005.08697/) (2020)
- [Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithms](https://mlsanity.com/paper/2010.12664/) (2020)
- [Incremental Learning-to-Learn with Statistical Guarantees](https://mlsanity.com/paper/1803.08089/) (2018)
- [Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization](https://mlsanity.com/paper/2011.02872/) (2020)
- [Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergence](https://mlsanity.com/paper/2010.09484/) (2020)
- [Theoretical bounds on estimation error for meta-learning](https://mlsanity.com/paper/2010.07140/) (2020)
- [Conditional Mutual Information-Based Generalization Bound for Meta Learning](https://mlsanity.com/paper/2010.10886/) (2020)

## Built on by (most cited first, 10 of 10)

- [Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning](https://mlsanity.com/paper/2106.00252/) (2021)
- [Learning to Learn Weight Generation via Local Consistency Diffusion](https://mlsanity.com/paper/2502.01117/) (2025)
- [Is Meta-Learning Out? Rethinking Unsupervised Few-Shot Classification with Limited Entropy](https://mlsanity.com/paper/2509.13185/) (2025)
- [Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios](https://mlsanity.com/paper/2210.14254/) (2022)
- [On the ERM Principle in Meta-Learning](https://mlsanity.com/paper/2411.17898/) (2024)
- [A Theory for Conditional Generative Modeling on Multiple Data Sources](https://mlsanity.com/paper/2502.14583/) (2025)
- [Time Associated Meta Learning for Clinical Prediction](https://mlsanity.com/paper/2303.02570/) (2023)
- [Unsupervised Meta-Learning via Dynamic Head and Heterogeneous Task Construction for Few-Shot Classification](https://mlsanity.com/paper/2410.02267/) (2024)
- [Bayesian Meta-Learning with Expert Feedback for Task-Shift Adaptation through Causal Embeddings](https://mlsanity.com/paper/2602.19788/) (2026)
- [Meta-RL with Shared Representations Enables Fast Adaptation in Energy Systems](https://mlsanity.com/paper/2603.08418/) (2026)

## Abstract (arXiv)

Meta-learning aims at optimizing the hyperparameters of a model class or training algorithm from the observation of data from a number of related tasks. Following the setting of Baxter [1], the tasks are assumed to belong to the same task environment, which is defined by a distribution over the space of tasks and by per-task data distributions. The statistical properties of the task environment thus dictate the similarity of the tasks. The goal of the meta-learner is to ensure that the hyperparameters obtain a small loss when applied for training of a new task sampled from the task environment. The difference between the resulting average loss, known as meta-population loss, and the corresponding empirical loss measured on the available data from related tasks, known as meta-generalization gap, is a measure of the generalization capability of the meta-learner. In this paper, we present novel information-theoretic bounds on the average absolute value of the meta-generalization gap. Unlike prior work [2], our bounds explicitly capture the impact of task relatedness, the number of tasks, and the number of data samples per task on the meta-generalization gap. Task similarity is gauged via the Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences. We illustrate the proposed bounds on the example of ridge regression with meta-learned bias.