mlsanityData to 7 Oct 2026

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

arXiv 2105.14099 ↗First posted 28 May 2021cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain performance improvements in the few-shot learning setting, where the number of training examples in the target tasks is severely limited. This gap originates from an assumption in the existing theories which supposes that the number of training examples in the observed tasks and the number of training examples in the target tasks follow the same distribution, an assumption that rarely holds in practice. By relaxing this assumption, we develop two PAC-Bayesian bounds tailored for the few-shot learning setting and show that two existing meta-learning algorithms (MAML and Reptile) can be derived from our bounds, thereby bridging the gap between practice and PAC-Bayesian theories. Furthermore, we derive a new computationally-efficient PACMAML algorithm, and show it outperforms existing meta-learning algorithms on several few-shot benchmark datasets.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

20 of 30 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Matching Networks for One Shot Learningcited by 2,246
  7. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  8. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  9. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  10. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  11. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  12. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
8 more
  1. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  2. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  3. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  4. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  5. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  6. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  7. 2015A New PAC-Bayesian Perspective on Domain Adaptationcited by 29
  8. 2017Cold-Start Reinforcement Learning with Softmax Policy Gradientcited by 15

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 10; 2023: 5; 2024: 3; 2025: 2.

21 papers.

Sort:

  1. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  2. 2022Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?cited by 13
  3. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  4. 2022Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practicecited by 11
  5. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  6. 2024More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithmscited by 10
  7. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  8. 2023Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayescited by 8
  9. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
  10. 2022Hardness-guided domain adaptation to recognise biomedical named entities under low-resource scenarioscited by 6
  11. 2022Improving Robust Generalization by Direct PAC-Bayesian Bound Minimizationcited by 6
  12. 2025From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learningcited by 5
9 more
  1. 2023Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisationcited by 3
  2. 2022A Statistical Model for Predicting Generalization in Few-Shot Classificationcited by 2
  3. 2023Federated Learning with Nonvacuous Generalisation Boundscited by 2
  4. 2024Tighter Generalisation Bounds via Interpolationcited by 2
  5. 2025Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizescited by 2
  6. 2024Task Attribute Distance for Few-Shot Learning: Theoretical Analysis and Applicationscited by 1
  7. 2022Hypernetwork approach to Bayesian MAMLcited by 0
  8. 2022Learning New Tasks from a Few Examples with Soft-Label Prototypescited by 0
  9. 2023Bayesian Flow Networks in Continual Learningcited by 0