mlsanityData to 14 Oct 2026

A PAC-Bayesian Tutorial with A Dropout Bound

arXiv 1307.2118 ↗First posted 8 Jul 2013cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

This tutorial gives a concise overview of existing PAC-Bayesian theory focusing on three generalization bounds. The first is an Occam bound which handles rules with finite precision parameters and which states that generalization loss is near training loss when the number of bits needed to write the rule is small compared to the sample size. The second is a PAC-Bayesian bound providing a generalization guarantee for posterior distributions rather than for individual rules. The PAC-Bayesian bound naturally handles infinite precision rule parameters, L2 regularization, em provides a bound for dropout training, and defines a natural notion of a single distinguished PAC-Bayesian posterior distribution. The third bound is a training-variance bound --- a kind of bias-variance analysis but with bias replaced by expected training loss. The training-variance bound dominates the other bounds but is more difficult to interpret. It seems to suggest variance reduction methods such as bagging and may ultimately provide a more meaningful analysis of dropouts.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Jul 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 11; 2023: 10; 2024: 5; 2025: 2; 2026: 2.

43 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  2. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  3. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  4. 2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 40
  5. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  6. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 30
  7. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  8. 2022Implicit regularization of dropoutcited by 25
  9. 2021How Tight Can PAC-Bayes be in the Small Data Regime?cited by 24
  10. 2021Information Complexity and Generalization Boundscited by 14
  11. 2023Online-to-PAC Conversions: Generalization Bounds via Regret Analysiscited by 14
  12. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
31 more
  1. 2021Learning Augmentation Distributions using Transformed Risk Minimizationcited by 13
  2. 2022PAC-Bayesian Domain Adaptation Bounds for Multiclass Learnerscited by 13
  3. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  4. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  5. 2021Robustness to Pruning Predicts Generalization in Deep Neural Networkscited by 10
  6. 2024Is K-fold cross validation the best model selection method for Machine Learning?cited by 10
  7. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimationcited by 8
  8. 2021Structured Dropout Variational Inference for Bayesian Neural Networkscited by 7
  9. 2022PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparisoncited by 7
  10. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  11. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
  12. 2021What training reveals about neural network complexitycited by 5
  13. 2021Task-Driven Detection of Distribution Shifts with Statistical Guarantees for Robot Learningcited by 4
  14. 2021PAC Bayesian Performance Guarantees for Deep (Stochastic) Networks in Medical Imagingcited by 3
  15. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  16. 2022A General framework for PAC-Bayes Bounds for Meta-Learningcited by 3
  17. 2024Tighter Risk Bounds for Mixtures of Expertscited by 3
  18. 2021Dropout in Training Neural Networks: Flatness of Solution and Noise Structurecited by 2
  19. 2023Generalization Error Bounds for Noisy, Iterative Algorithms via Maximal Leakagecited by 2
  20. 2024Tighter Generalisation Bounds via Interpolationcited by 2
  21. 2021Stronger Generalization Guarantees for Robot Learning by Combining Generative Models and Real-World Datacited by 1
  22. 2022Multi-class Classification with Fuzzy-feature Observations: Theory and Algorithmscited by 1
  23. 2023Practicality of generalization guarantees for unsupervised domain adaptation with neural networkscited by 1
  24. 2024Domain Generalization Guided by Large-Scale Pre-Trained Priorscited by 1
  25. 2025Generalization Bounds for Quantum Learning via R'enyi Divergencescited by 1
  26. 2026PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priorscited by 1
  27. 2023Improving Generalization in Meta-Learning via Meta-Gradient Augmentationcited by 0
  28. 2023Generalization within in silico screeningcited by 0
  29. 2023Feature Noise Boosts DNN Generalization under Label Noisecited by 0
  30. 2024Statistical Agnostic Regression: a machine learning method to validate regression modelscited by 0
  31. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Modelscited by 0