mlsanityData to 7 Oct 2026

Non-Vacuous Generalisation Bounds for Shallow Neural Networks

arXiv 2202.01627 ↗First posted 3 Feb 2022cs.LG stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

We focus on a specific class of shallow neural networks with a single hidden layer, namely those with L2-normalised data and either a sigmoid-shaped Gaussian error function ("erf") activation or a Gaussian Error Linear Unit (GELU) activation. For these networks, we derive new generalisation bounds through the PAC-Bayesian theory; unlike most existing such bounds they apply to neural networks with deterministic rather than randomised parameters. Our bounds are empirically non-vacuous when the network is trained with vanilla stochastic gradient descent on MNIST and Fashion-MNIST.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 27 of its 55 references that are in the map. First posted on arXiv 3 Feb 2022.

Builds on

27 of 55 references.

References · 27

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  4. 2017Exploring Generalization in Deep Learningcited by 595
  5. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  9. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  10. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  11. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  12. 2017Generalization in Deep Learningcited by 146
15 more
  1. 2019A Primer on PAC-Bayesian Learningcited by 130
  2. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  3. 2020Tighter risk certificates for neural networkscited by 112
  4. 2017Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priorscited by 81
  5. 2020In Search of Robust Measures of Generalizationcited by 79
  6. 2020On the role of data in PAC-Bayes boundscited by 64
  7. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  8. 2020NeurIPS 2020 Competition: Predicting Generalization in Deep Learningcited by 41
  9. 2019Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networkscited by 38
  10. 2020Second Order PAC-Bayesian Bounds for the Weighted Majority Votecited by 34
  11. 2020Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networkscited by 29
  12. 2018PAC-Bayes bounds for stable algorithms with instance-dependent priorscited by 21
  13. 2021Learning PAC-Bayes Priors for Probabilistic Neural Networkscited by 21
  14. 2021On Margins and Derandomisation in PAC-Bayescited by 19
  15. 2021Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Boundcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 9; 2024: 2; 2025: 2; 2026: 2.

22 papers.

Sort:

  1. 2023MMD-FUSE: Learning and Combining Kernels for Two-Sample Testing Without Data Splittingcited by 33
  2. 2023A unified recipe for deriving (time-uniform) PAC-Bayes boundscited by 25
  3. 2022PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingalescited by 23
  4. 2022A PAC-Bayesian Generalization Bound for Equivariant Networkscited by 17
  5. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  6. 2022Generalisation under gradient descent via deterministic PAC-Bayescited by 9
  7. 2022Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficultycited by 9
  8. 2022On Margins and Generalisation for Voting Classifierscited by 7
  9. 2023Comparing Comparators in Generalization Boundscited by 5
  10. 2023Generalization Bounds for Neural Belief Propagation Decoderscited by 4
  11. 2024Gentle Local Robustness implies Generalizationcited by 4
  12. 2022Controlling Multiple Errors Simultaneously with a PAC-Bayes Boundcited by 3
10 more
  1. 2023Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisationcited by 3
  2. 2022A Note on the Efficient Evaluation of PAC-Bayes Boundscited by 2
  3. 2023Federated Learning with Nonvacuous Generalisation Boundscited by 2
  4. 2025Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained modelscited by 2
  5. 2023A note on regularised NTK dynamics with an application to PAC-Bayesian trainingcited by 1
  6. 2025Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?cited by 1
  7. 2026PAC-Bayesian Reconstruction Guarantees for Time Series Variational Autoencoderscited by 1
  8. 2023The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networkscited by 0
  9. 2024A Generalization Bound for Nearly-Linear Networkscited by 0
  10. 2026Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalitiescited by 0