mlsanityData to 7 Oct 2026

Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes

arXiv 2302.11709 ↗First posted 23 Feb 2023stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Bernstein's condition is a key assumption that guarantees fast rates in machine learning. For example, the Gibbs algorithm with prior π has an excess risk in O(dπ/n), as opposed to the standard O(√dπ/n), where n denotes the number of observations and dπ is a complexity parameter which depends on the prior π. In this paper, we examine the Gibbs algorithm in the context of meta-learning, i.e., when learning the prior π from T tasks (with n observations each) generated by a meta distribution. Our main result is that Bernstein's condition always holds at the meta level, regardless of its validity at the observation level. This implies that the additional cost to learn the Gibbs prior π, which will reduce the term dπ across tasks, is in O(1/T), instead of the expected O(1/√T). We further illustrate how this result improves on standard rates in three different settings: discrete priors, Gaussian priors and mixture of Gaussians priors.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 57 of its 98 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

57 of 98 references.

References · 57

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  3. 2016Matching Networks for One Shot Learningcited by 2,246
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  6. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  7. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  8. 2017A Survey on Multi-Task Learningcited by 838
  9. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  10. 2019Meta-Learning with Implicit Gradientscited by 523
  11. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  12. 2016Learning to reinforcement learncited by 369
38 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2017Meta Networkscited by 268
  3. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  4. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  5. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  6. 2015The Benefit of Multitask Representation Learningcited by 194
  7. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  8. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  9. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  10. 2013A General Framework for Updating Belief Distributionscited by 172
  11. 2020Provable Meta-Learning of Linear Representationscited by 161
  12. 2019Online Meta-Learningcited by 137
  13. 2019A Primer on PAC-Bayesian Learningcited by 130
  14. 2017Few-Shot Image Recognition by Predicting Parameters from Activationscited by 122
  15. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  16. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  17. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  18. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  19. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  20. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  21. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  22. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  23. 2019Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Taskscited by 48
  24. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  25. 2019Domain Adaptive Dialog Generation via Meta Learningcited by 41
  26. 2020The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuningcited by 40
  27. 2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
  28. 2018Natural Language to Structured Query Generation via Meta-Learningcited by 33
  29. 2020Information-Theoretic Generalization Bounds for Meta-Learning and Applicationscited by 32
  30. 2020How Important is the Train-Validation Split in Meta-Learning?cited by 30
  31. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 29
  32. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  33. 2016Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayescited by 27
  34. 2019Uncertainty in Model-Agnostic Meta-Learning using Variational Inferencecited by 23
  35. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  36. 2019Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processescited by 20
  37. 2016Regret Bounds for Lifelong Learningcited by 19
  38. 2018Incremental Learning-to-Learn with Statistical Guaranteescited by 16

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 2; 2026: 1.

8 papers.

  1. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  2. 2024More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithmscited by 10
  3. 2023Meta-learning Control Variates: Variance Reduction with Limited Datacited by 6
  4. 2025From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learningcited by 5
  5. 2023Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisationcited by 3
  6. 2024On Uncertainty Quantification for Near-Bayes Optimal Algorithmscited by 3
  7. 2025Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizescited by 2
  8. 2026Learning to Adapt Cross-Domain Preferences via Meta-LoRA for LLM Personalizationcited by 0