mlsanityData to 14 Oct 2026

On the Importance of Gradient Norm in PAC-Bayesian Bounds

arXiv 2210.06143 ↗First posted 12 Oct 2022cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Generalization bounds which assess the difference between the true risk and the empirical risk, have been studied extensively. However, to obtain bounds, current techniques use strict assumptions such as a uniformly bounded or a Lipschitz loss function. To avoid these assumptions, in this paper, we follow an alternative approach: we relax uniform bounds assumptions by using on-average bounded loss and on-average bounded gradient norm assumptions. Following this relaxation, we propose a new generalization bound that exploits the contractivity of the log-Sobolev inequalities. These inequalities add an additional loss-gradient norm term to the generalization bound, which is intuitively a surrogate of the model complexity. We apply the proposed bound on Bayesian deep nets and empirically analyze the effect of this new loss-gradient norm term on different neural architectures.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 47 of its 76 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

47 of 76 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  10. 2015Weight Uncertainty in Neural Networkscited by 554
  11. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  12. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
35 more
  1. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  2. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  3. 2018How Does Batch Normalization Help Optimization?cited by 357
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  5. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  6. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  7. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  8. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  9. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  10. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  11. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  12. 2020Tighter risk certificates for neural networkscited by 112
  13. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  14. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  15. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 88
  16. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  17. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  18. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  19. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  20. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  21. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  22. 2020On the role of data in PAC-Bayes boundscited by 64
  23. 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
  24. 2014Learning without Concentrationcited by 50
  25. 2016Simpler PAC-Bayesian Bounds for Hostile Datacited by 46
  26. 2013A PAC-Bayesian Tutorial with A Dropout Boundcited by 43
  27. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  28. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 30
  29. 2017A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descentcited by 29
  30. 2021How Tight Can PAC-Bayes be in the Small Data Regime?cited by 24
  31. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  32. 2019Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processescited by 20
  33. 2019PAC-Bayes under potentially heavy tailscited by 16
  34. 2017A Tight Excess Risk Bound via a Unified PAC-Bayesian-Rademacher-Shtarkov-MDL Complexitycited by 7
  35. 2013Horizon-Independent Optimal Prediction with Log-Loss in Exponential Familiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 0; 2026: 1.

6 papers.

  1. 2023Layer Collaboration in the Forward-Forward Algorithmcited by 21
  2. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  3. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  4. 2024PAC-Bayes-Chernoff bounds for unbounded lossescited by 9
  5. 2022Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networkscited by 0
  6. 2026A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learningcited by 0