# A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning

arXiv: https://arxiv.org/abs/2106.15615 · HTML: https://mlsanity.com/paper/2106.15615/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 30 of 30)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Few-Shot Learning via Learning the Representation, Provably](https://mlsanity.com/paper/2002.09434/) (2020)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms](https://mlsanity.com/paper/1908.10400/) (2019)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [A PAC-Bayesian bound for Lifelong Learning](https://mlsanity.com/paper/1311.2838/) (2013)
- [Provable Guarantees for Gradient-Based Meta-Learning](https://mlsanity.com/paper/1902.10644/) (2019)
- [Torchmeta: A Meta-Learning library for PyTorch](https://mlsanity.com/paper/1909.06576/) (2019)
- [Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks](https://mlsanity.com/paper/2002.06753/) (2020)
- [Meta-learning for mixed linear regression](https://mlsanity.com/paper/2002.08936/) (2020)
- [How Important is the Train-Validation Split in Meta-Learning?](https://mlsanity.com/paper/2010.05843/) (2020)
- [Regret Bounds for Lifelong Learning](https://mlsanity.com/paper/1610.08628/) (2016)
- [A Sample Complexity Separation between Non-Convex and Convex Meta-Learning](https://mlsanity.com/paper/2002.11172/) (2020)
- [Incremental Learning-to-Learn with Statistical Guarantees](https://mlsanity.com/paper/1803.08089/) (2018)
- [Guarantees for Tuning the Step Size using a Learning-to-Learn Approach](https://mlsanity.com/paper/2006.16495/) (2020)
- [Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets](https://mlsanity.com/paper/2006.14606/) (2020)
- [BOIL: Towards Representation Change for Few-shot Learning](https://mlsanity.com/paper/2008.08882/) (2020)

## Built on by (most cited first, 12 of 12)

- [Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2109.14595/) (2021)
- [Identification of Negative Transfers in Multitask Learning Using Surrogate Models](https://mlsanity.com/paper/2303.14582/) (2023)
- [Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2307.06887/) (2023)
- [Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data](https://mlsanity.com/paper/2308.04428/) (2023)
- [Understanding Benign Overfitting in Gradient-Based Meta Learning](https://mlsanity.com/paper/2206.13482/) (2022)
- [ALP: Data Augmentation using Lexicalized PCFGs for Few-Shot Text Classification](https://mlsanity.com/paper/2112.11916/) (2021)
- [On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift](https://mlsanity.com/paper/2312.15551/) (2023)
- [Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance](https://mlsanity.com/paper/2201.12692/) (2022)
- [Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning](https://mlsanity.com/paper/2304.04312/) (2023)
- [Meta-free few-shot learning via representation learning with weight averaging](https://mlsanity.com/paper/2204.12466/) (2022)
- [First-order ANIL provably learns representations despite overparametrization](https://mlsanity.com/paper/2303.01335/) (2023)
- [Robust and Adaptive Spectral Method for Representation Multi-Task Learning with Contamination](https://mlsanity.com/paper/2509.06575/) (2025)

## Abstract (arXiv)

An effective approach in meta-learning is to utilize multiple "train tasks" to learn a good initialization for model parameters that can help solve unseen "test tasks" with very few samples by fine-tuning from this initialization. Although successful in practice, theoretical understanding of such methods is limited. This work studies an important aspect of these methods: splitting the data from each task into train (support) and validation (query) sets during meta-training. Inspired by recent work (Raghu et al., 2020), we view such meta-learning methods through the lens of representation learning and argue that the train-validation split encourages the learned representation to be low-rank without compromising on expressivity, as opposed to the non-splitting variant that encourages high-rank representations. Since sample efficiency benefits from low-rankness, the splitting strategy will require very few samples to solve unseen test tasks. We present theoretical results that formalize this idea for linear representation learning on a subspace meta-learning instance, and experimentally verify this practical benefit of splitting in simulations and on standard meta-learning benchmarks.