# How Important is the Train-Validation Split in Meta-Learning?

arXiv: https://arxiv.org/abs/2010.05843 · HTML: https://mlsanity.com/paper/2010.05843/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 30 of 30)

- [Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation](https://mlsanity.com/paper/2106.09017/) (2021)
- [AutoBalance: Optimized Loss Functions for Imbalanced Data](https://mlsanity.com/paper/2201.01212/) (2022)
- [Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2109.14595/) (2021)
- [How Fine-Tuning Allows for Effective Meta-Learning](https://mlsanity.com/paper/2105.02221/) (2021)
- [Betty: An Automatic Differentiation Library for Multilevel Optimization](https://mlsanity.com/paper/2207.02849/) (2022)
- [Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning](https://mlsanity.com/paper/2203.09137/) (2022)
- [Generalization Guarantees for Neural Architecture Search with Train-Validation Split](https://mlsanity.com/paper/2104.14132/) (2021)
- [Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?](https://mlsanity.com/paper/2203.03059/) (2022)
- [A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning](https://mlsanity.com/paper/2106.15615/) (2021)
- [Meta-learning with negative learning rates](https://mlsanity.com/paper/2102.00940/) (2021)
- [Understanding Benign Overfitting in Gradient-Based Meta Learning](https://mlsanity.com/paper/2206.13482/) (2022)
- [ALP: Data Augmentation using Lexicalized PCFGs for Few-Shot Text Classification](https://mlsanity.com/paper/2112.11916/) (2021)
- [A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigm](https://mlsanity.com/paper/2310.00708/) (2023)
- [Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution](https://mlsanity.com/paper/2102.11503/) (2021)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [Provable Generalization of Overparameterized Meta-learning Trained with SGD](https://mlsanity.com/paper/2206.09136/) (2022)
- [Meta-learning in healthcare: A survey](https://mlsanity.com/paper/2308.02877/) (2023)
- [Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes](https://mlsanity.com/paper/2301.06806/) (2023)
- [Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes](https://mlsanity.com/paper/2302.11709/) (2023)
- [How to distribute data across tasks for meta-learning?](https://mlsanity.com/paper/2103.08463/) (2021)
- [A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning](https://mlsanity.com/paper/2106.14749/) (2021)
- [Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance](https://mlsanity.com/paper/2201.12692/) (2022)
- [Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning](https://mlsanity.com/paper/2304.04312/) (2023)
- [BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2410.09758/) (2024)
- [Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning](https://mlsanity.com/paper/2410.22788/) (2024)
- [Learning Tensor Representations for Meta-Learning](https://mlsanity.com/paper/2201.07348/) (2022)
- [Meta-RegGNN: Predicting Verbal and Full-Scale Intelligence Scores using Graph Neural Networks and Meta-Learning](https://mlsanity.com/paper/2209.13530/) (2022)
- [TaskMix: Data Augmentation for Meta-Learning of Spoken Intent Understanding](https://mlsanity.com/paper/2210.06341/) (2022)
- [Designing Time-Series Models With Hypernetworks & Adversarial Portfolios](https://mlsanity.com/paper/2407.20352/) (2024)
- [Task-Level Contrastiveness for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2510.03509/) (2025)

## Abstract (arXiv)

Meta-learning aims to perform fast adaptation on a new task through learning a "prior" from multiple existing tasks. A common practice in meta-learning is to perform a train-validation split (train-val method) where the prior adapts to the task on one split of the data, and the resulting predictor is evaluated on another split. Despite its prevalence, the importance of the train-validation split is not well understood either in theory or in practice, particularly in comparison to the more direct train-train method, which uses all the per-task data for both training and evaluation. We provide a detailed theoretical study on whether and when the train-validation split is helpful in the linear centroid meta-learning problem. In the agnostic case, we show that the expected loss of the train-val method is minimized at the optimal prior for meta testing, and this is not the case for the train-train method in general without structural assumptions on the data. In contrast, in the realizable case where the data are generated from linear models, we show that both the train-val and train-train losses are minimized at the optimal prior in expectation. Further, perhaps surprisingly, our main result shows that the train-train method achieves a strictly better excess loss in this realizable case, even when the regularization parameter and split ratio are optimally tuned for both methods. Our results highlight that sample splitting may not always be preferable, especially when the data is realizable by the model. We validate our theories by experimentally showing that the train-train method can indeed outperform the train-val method, on both simulations and real meta-learning tasks.