mlsanityData to 7 Oct 2026

Provable Generalization of Overparameterized Meta-learning Trained with SGD

arXiv 2206.09136 ↗First posted 18 Jun 2022cs.LG math.ST stat.THcited by 9

Not on the map. Why?

Abstract (arXiv)

Despite the superior empirical success of deep meta-learning, theoretical understanding of overparameterized meta-learning is still limited. This paper studies the generalization of a widely used meta-learning approach, Model-Agnostic Meta-Learning (MAML), which aims to find a good initialization for fast adaptation to new tasks. Under a mixed linear regression model, we analyze the generalization properties of MAML trained with SGD in the overparameterized regime. We provide both upper and lower bounds for the excess risk of MAML, which captures how SGD dynamics affect these generalization bounds. With such sharp characterizations, we further explore how various learning parameters impact the generalization capability of overparameterized MAML, including explicitly identifying typical data and task distributions that can achieve diminishing generalization error with overparameterization, and characterizing the impact of adaptation learning rate on both excess risk and the early stopping time. Our theoretical findings are further validated by experiments.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 18 Jun 2022.

Builds on

43 of 57 references.

References · 43

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2019Meta-Learning with Implicit Gradientscited by 523
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  7. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  8. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  9. 2018How to train your MAMLcited by 306
  10. 2018Meta-Learning: A Surveycited by 254
  11. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  12. 2020Benign overfitting in ridge regressioncited by 152
31 more
  1. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  2. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  3. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  4. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  5. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  6. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  7. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  8. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  9. 2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
  10. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  11. 2019Few-shot Text Classification with Distributional Signaturescited by 58
  12. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  13. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  14. 2020Meta-learning for mixed linear regressioncited by 44
  15. 2020Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Modelcited by 39
  16. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 37
  17. 2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
  18. 2020Information-Theoretic Generalization Bounds for Meta-Learning and Applicationscited by 32
  19. 2020How Important is the Train-Validation Split in Meta-Learning?cited by 30
  20. 2022MAML and ANIL Provably Learn Representationscited by 30
  21. 2020Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learningcited by 28
  22. 2017A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)cited by 25
  23. 2019Beating SGD Saturation with Tail-Averaging and Minibatchingcited by 21
  24. 2020On the Global Optimality of Model-Agnostic Meta-Learningcited by 21
  25. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  26. 2020Modeling and Optimization Trade-off in Meta-learningcited by 13
  27. 2022Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?cited by 13
  28. 2013Accelerated Mini-Batch Stochastic Dual Coordinate Ascentcited by 11
  29. 2021Meta-learning with negative learning ratescited by 11
  30. 2022Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regimecited by 8
  31. 2020Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Netscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 5; 2025: 1; 2026: 1.

9 papers.

  1. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  2. 2024Meta-Learning with Generalized Ridge Regression: High-dimensional Asymptotics, Optimality and Hyper-covariance Estimationcited by 5
  3. 2023Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learningcited by 4
  4. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
  5. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparatorcited by 2
  6. 2025FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fieldscited by 1
  7. 2024Intelligent Chemical Purification Technique Based on Machine Learningcited by 0
  8. 2024Dynamics of Meta-learning Representation in the Teacher-student Scenariocited by 0
  9. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0