# Information Theoretic Meta Learning with Gaussian Processes

arXiv: https://arxiv.org/abs/2009.03228 · HTML: https://mlsanity.com/paper/2009.03228/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 11 of 11)

- [Non-Gaussian Gaussian Processes for Few-Shot Regression](https://mlsanity.com/paper/2110.13561/) (2021)
- [ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning](https://mlsanity.com/paper/2109.13305/) (2021)
- [Bayesian Meta-Learning Through Variational Gaussian Processes](https://mlsanity.com/paper/2110.11044/) (2021)
- [Information-theoretic Analysis of Test Data Sensitivity in Uncertainty](https://mlsanity.com/paper/2307.12456/) (2023)
- [Federated Neural Nonparametric Point Processes](https://mlsanity.com/paper/2410.05637/) (2024)
- [Towards a population-informed approach to the definition of data-driven models for structural dynamics](https://mlsanity.com/paper/2307.09862/) (2023)
- [Accelerating Convergence in Bayesian Few-Shot Classification](https://mlsanity.com/paper/2405.01507/) (2024)
- [On the ERM Principle in Meta-Learning](https://mlsanity.com/paper/2411.17898/) (2024)
- [Meta-learning Amidst Heterogeneity and Ambiguity](https://mlsanity.com/paper/2107.02228/) (2021)
- [Multi-Task Neural Processes](https://mlsanity.com/paper/2110.14953/) (2021)
- [A Meta-Learning Approach to Population-Based Modelling of Structures](https://mlsanity.com/paper/2302.07980/) (2023)

## Abstract (arXiv)

We formulate meta learning using information theoretic concepts; namely, mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. This framework unifies existing gradient-based algorithms and also allows us to derive new algorithms. In particular, we develop a memory-based algorithm that uses Gaussian processes to obtain non-parametric encoding representations. We demonstrate our method on a few-shot regression problem and on four few-shot classification problems, obtaining competitive accuracy when compared to existing baselines.