mlsanityData to 7 Oct 2026

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

arXiv 2002.05551 ↗First posted 13 Feb 2020stat.ML cs.LGcited by 85

Not on the map. Why?

Abstract (arXiv)

Meta-learning can successfully acquire useful inductive biases from data. Yet, its generalization properties to unseen learning tasks are poorly understood. Particularly if the number of meta-training tasks is small, this raises concerns about overfitting. We provide a theoretical analysis using the PAC-Bayesian framework and derive novel generalization bounds for meta-learning. Using these bounds, we develop a class of PAC-optimal meta-learning algorithms with performance guarantees and a principled meta-level regularization. Unlike previous PAC-Bayesian meta-learners, our method results in a standard stochastic optimization problem which can be solved efficiently and scales well. When instantiating our PAC-optimal hyper-posterior (PACOH) with Gaussian processes and Bayesian Neural Networks as base learners, the resulting methods yield state-of-the-art performance, both in terms of predictive accuracy and the quality of uncertainty estimates. Thanks to their principled treatment of uncertainty, our meta-learners can also be successfully employed for sequential decision problems.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 20; 2023: 19; 2024: 17; 2025: 4; 2026: 5.

85 papers.

Sort:

By area:

  1. 2021Transformers Can Do Bayesian Inferencecited by 338
  2. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  3. 2021Bayesian Neural Network Priors Revisitedcited by 121
  4. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  5. 2024Vanilla Bayesian Optimization Performs Great in High Dimensionscited by 87
  6. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  7. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  8. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  9. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  10. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 31
  11. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  12. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 29
38 more
  1. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  2. 2021How Tight Can PAC-Bayes be in the Small Data Regime?cited by 24
  3. 2021Meta-Learning Reliable Priors in the Function Spacecited by 23
  4. 2023A General Framework for User-Guided Bayesian Optimizationcited by 22
  5. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  6. 2023PeFLL: Personalized Federated Learning by Learning to Learncited by 21
  7. 2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
  8. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  9. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  10. 2022Meta-Learning Priors for Safe Bayesian Optimizationcited by 14
  11. 2022Modeling Human Exploration Through Resource-Rational Reinforcement Learningcited by 13
  12. 2022Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?cited by 13
  13. 2021Non-Gaussian Gaussian Processes for Few-Shot Regressioncited by 12
  14. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  15. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  16. 2022Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practicecited by 11
  17. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  18. 2023Metalearning with Very Few Samples Per Taskcited by 11
  19. 2021An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learningcited by 10
  20. 2021Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learningcited by 10
  21. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  22. 2024More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithmscited by 10
  23. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  24. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  25. 2023Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayescited by 8
  26. 2021Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stabilitycited by 7
  27. 2022PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparisoncited by 7
  28. 2021Meta-strategy for Learning Tuning Parameters with Guaranteescited by 6
  29. 2022Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approachcited by 6
  30. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
  31. 2022Improving Robust Generalization by Direct PAC-Bayesian Bound Minimizationcited by 6
  32. 2023Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learningcited by 6
  33. 2022Meta-Learning Hypothesis Spaces for Sequential Decision-makingcited by 5
  34. 2022MARS: Meta-Learning as Score Matching in the Function Spacecited by 5
  35. 2024Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approachcited by 5
  36. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  37. 2022PAC-Bayesian Lifelong Learning For Multi-Armed Banditscited by 4
  38. 2023DAC-MR: Data Augmentation Consistency Based Meta-Regularization for Meta-Learningcited by 4

50 most cited of 85 All 85 (JSON)