mlsanityData to 7 Oct 2026

Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability

arXiv 2102.06589 ↗First posted 12 Feb 2021cs.LG cs.AI stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

We are motivated by the problem of providing strong generalization guarantees in the context of meta-learning. Existing generalization bounds are either challenging to evaluate or provide vacuous guarantees in even relatively simple settings. We derive a probably approximately correct (PAC) bound for gradient-based meta-learning using two different generalization frameworks in order to deal with the qualitatively different challenges of generalization at the "base" and "meta" levels. We employ bounds for uniformly stable algorithms at the base level and bounds from the PAC-Bayes framework at the meta level. The result of this approach is a novel PAC bound that is tighter when the base learner adapts quickly, which is precisely the goal of meta-learning. We show that our bound provides a tighter guarantee than other bounds on a toy non-convex problem on the unit sphere and a text-based classification example. We also present a practical regularization scheme motivated by the bound in settings where the bound is loose and demonstrate improved performance over baseline techniques.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 38 of its 80 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

38 of 80 references.

References · 38

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Matching Networks for One Shot Learningcited by 2,246
  5. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  7. 2019Introduction to Online Convex Optimizationcited by 793
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
26 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  5. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  6. 2017A Rewriting System for Convex Optimization Problemscited by 178
  7. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  8. 2020Tighter risk certificates for neural networkscited by 112
  9. 2019Meta-Learning without Memorizationcited by 92
  10. 2019Sharper bounds for uniformly stable algorithmscited by 91
  11. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  12. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  13. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  14. 2020learn2learn: A Library for Meta-Learning Researchcited by 69
  15. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  16. 2020PAC-Bayes Analysis Beyond the Usual Boundscited by 63
  17. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  18. 2013A PAC-Bayesian Tutorial with A Dropout Boundcited by 43
  19. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  20. 2019PAC-Bayes with Backpropcited by 36
  21. 2017A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descentcited by 29
  22. 2018PAC-Bayes bounds for stable algorithms with instance-dependent priorscited by 21
  23. 2016Regret Bounds for Lifelong Learningcited by 19
  24. 2016Stability revisited: new generalisation bounds for the Leave-one-Outcited by 10
  25. 2019An Exponential Efron-Stein Inequality for Lq Stable Learning Rulescited by 6
  26. 2013On Learnability, Complexity and Stabilitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 7; 2024: 6; 2025: 5; 2026: 1.

29 papers.

Sort:

  1. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  2. 2023Exponential Smoothing for Off-Policy Learningcited by 15
  3. 2022Meta-Learning Priors for Safe Bayesian Optimizationcited by 14
  4. 2022Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?cited by 13
  5. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  6. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  7. 2022Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practicecited by 11
  8. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  9. 2024More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithmscited by 10
  10. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  11. 2023Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayescited by 8
  12. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
17 more
  1. 2024Tight PAC-Bayesian Risk Certificates for Contrastive Learningcited by 6
  2. 2024Meta-Learning with Generalized Ridge Regression: High-dimensional Asymptotics, Optimality and Hyper-covariance Estimationcited by 5
  3. 2024Data-Driven Performance Guarantees for Classical and Learned Optimizerscited by 5
  4. 2025From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learningcited by 5
  5. 2025Is Meta-Learning Out? Rethinking Unsupervised Few-Shot Classification with Limited Entropycited by 4
  6. 2023PAC-Bayesian Soft Actor-Critic Learningcited by 3
  7. 2023Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisationcited by 3
  8. 2022A Statistical Model for Predicting Generalization in Few-Shot Classificationcited by 2
  9. 2023Federated Learning with Nonvacuous Generalisation Boundscited by 2
  10. 2024Tighter Generalisation Bounds via Interpolationcited by 2
  11. 2024Meta-learning for Positive-unlabeled Classificationcited by 2
  12. 2025Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizescited by 2
  13. 2022Hypernetwork approach to Bayesian MAMLcited by 0
  14. 2023Towards Understanding the Generalizability of Delayed Stochastic Gradient Descentcited by 0
  15. 2025Domain-Generalization to Improve Learning in Meta-Learning Algorithmscited by 0
  16. 2025An Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learningcited by 0
  17. 2026On the Stability and Generalization of First-order Bilevel Minimax Optimizationcited by 0