# On the ERM Principle in Meta-Learning

arXiv: https://arxiv.org/abs/2411.17898 · HTML: https://mlsanity.com/paper/2411.17898/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Natural Language Processing: State of The Art, Current Trends and Challenges](https://mlsanity.com/paper/1708.05148/) (2017)
- [Minimax Analysis of Active Learning](https://mlsanity.com/paper/1410.0996/) (2014)
- [A Theory of Universal Learning](https://mlsanity.com/paper/2011.04483/) (2020)
- [Proper Learning, Helly Number, and an Optimal SVM Bound](https://mlsanity.com/paper/2005.11818/) (2020)
- [Natural Language Processing Advancements By Deep Learning: A Survey](https://mlsanity.com/paper/2003.01200/) (2020)
- [Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2109.14595/) (2021)
- [Refined Error Bounds for Several Learning Algorithms](https://mlsanity.com/paper/1512.07146/) (2015)
- [Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principle](https://mlsanity.com/paper/1502.06309/) (2015)
- [Information Theoretic Meta Learning with Gaussian Processes](https://mlsanity.com/paper/2009.03228/) (2020)
- [An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning](https://mlsanity.com/paper/2101.08390/) (2021)
- [Theoretical bounds on estimation error for meta-learning](https://mlsanity.com/paper/2010.07140/) (2020)
- [A Distribution-Dependent Analysis of Meta-Learning](https://mlsanity.com/paper/2011.00344/) (2020)
- [Convex Set Disjointness, Distributed Learning of Halfspaces, and LP Feasibility](https://mlsanity.com/paper/1909.03547/) (2019)

## Built on by (most cited first, 1 of 1)

- [How Many Domains Suffice for Domain Generalization? A Tight Characterization via the Domain Shattering Dimension](https://mlsanity.com/paper/2506.16704/) (2025)

## Abstract (arXiv)

Classic supervised learning involves algorithms trained on n labeled examples to produce a hypothesis h ∈ H aimed at performing well on unseen examples. Meta-learning extends this by training across n tasks, with m examples per task, producing a hypothesis class H within some meta-class mathbbH. This setting applies to many modern problems such as in-context learning, hypernetworks, and learning-to-learn. A common method for evaluating the performance of supervised learning algorithms is through their learning curve, which depicts the expected error as a function of the number of training examples. In meta-learning, the learning curve becomes a two-dimensional learning surface, which evaluates the expected error on unseen domains for varying values of n (number of tasks) and m (number of training examples). Our findings characterize the distribution-free learning surfaces of meta-Empirical Risk Minimizers when either m or n tend to infinity: we show that the number of tasks must increase inversely with the desired error. In contrast, we show that the number of examples exhibits very different behavior: it satisfies a dichotomy where every meta-class conforms to one of the following conditions: (i) either m must grow inversely with the error, or (ii) a finite number of examples per task suffices for the error to vanish as n goes to infinity. This finding illustrates and characterizes cases in which a small number of examples per task is sufficient for successful learning. We further refine this for positive values of varepsilon and identify for each varepsilon how many examples per task are needed to achieve an error of varepsilon in the limit as the number of tasks n goes to infinity. We achieve this by developing a necessary and sufficient condition for meta-learnability using a bounded number of examples per domain.