mlsanityData to 7 Oct 2026

Understanding Benign Overfitting in Gradient-Based Meta Learning

arXiv 2206.13482 ↗First posted 27 Jun 2022cs.LG math.OC stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Meta learning has demonstrated tremendous success in few-shot learning with limited supervised data. In those settings, the meta model is usually overparameterized. While the conventional statistical learning theory suggests that overparameterized models tend to overfit, empirical evidence reveals that overparameterized meta learning methods still work well -- a phenomenon often called "benign overfitting." To understand this phenomenon, we focus on the meta learning settings with a challenging bilevel structure that we term the gradient-based meta learning, and analyze its generalization performance under an overparameterized meta linear regression model. While our analysis uses the relatively tractable linear models, our theory contributes to understanding the delicate interplay among data heterogeneity, model adaptation and benign overfitting in gradient-based meta learning tasks. We corroborate our theoretical claims through numerical simulations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

39 of 54 references.

References · 39

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  3. 2016Matching Networks for One Shot Learningcited by 2,246
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  5. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  6. 2019A Closer Look at Few-shot Classificationcited by 679
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2016Learning to learn by gradient descent by gradient descentcited by 616
  9. 2019Benign Overfitting in Linear Regressioncited by 544
  10. 2019Meta-Learning with Implicit Gradientscited by 523
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
27 more
  1. 2020Benign overfitting in ridge regressioncited by 152
  2. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  3. 2019Harmless interpolation of noisy data in regressioncited by 121
  4. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  5. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  7. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  8. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  9. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  10. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  11. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  12. 2020Meta-learning for mixed linear regressioncited by 44
  13. 2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
  14. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  15. 2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
  16. 2020Information-Theoretic Generalization Bounds for Meta-Learning and Applicationscited by 32
  17. 2020How Important is the Train-Validation Split in Meta-Learning?cited by 30
  18. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 29
  19. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  20. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  21. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  22. 2020Modeling and Optimization Trade-off in Meta-learningcited by 13
  23. 2022Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?cited by 13
  24. 2021A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learningcited by 12
  25. 2021Meta-learning with negative learning ratescited by 11
  26. 2021Benign Overfitting in Adversarially Robust Linear Classificationcited by 11
  27. 2022Provable Generalization of Overparameterized Meta-learning Trained with SGDcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 5; 2025: 2; 2026: 1.

11 papers.

  1. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  2. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  3. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16
  4. 2023Meta-learning in healthcare: A surveycited by 9
  5. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
  6. 2024Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Expertscited by 3
  7. 2025Scanning Trojaned Models Using Out-of-Distribution Samplescited by 3
  8. 2024Probability Distribution Learning and Its Application in Deep Learningcited by 1
  9. 2024Neuromodulated Meta-Learningcited by 0
  10. 2025Revisiting Meta-Learning with Noisy Labels: Reweighting Dynamics and Theoretical Guaranteescited by 0
  11. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0