mlsanityData to 7 Oct 2026

Wide stochastic networks: Gaussian limit and PAC-Bayesian training

arXiv 2106.09798 ↗First posted 17 Jun 2021stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

The limit of infinite width allows for substantial simplifications in the analytical study of over-parameterised neural networks. With a suitable random initialisation, an extremely large network exhibits an approximately Gaussian behaviour. In the present work, we establish a similar result for a simple stochastic architecture whose parameters are random variables, holding both before and during training. The explicit evaluation of the output distribution allows for a PAC-Bayesian training procedure that directly optimises the generalisation bound. For a large but finite-width network, we show empirically on MNIST that this training approach can outperform standard PAC-Bayesian methods.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

19 of 40 references.

References · 19

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  5. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  8. 2016Deep Information Propagationcited by 194
  9. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  10. 2019A Primer on PAC-Bayesian Learningcited by 130
  11. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  12. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
7 more
  1. 2020Tighter risk certificates for neural networkscited by 112
  2. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  3. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  4. 2020Stable ResNetcited by 49
  5. 2019Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networkscited by 38
  6. 2020Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networkscited by 29
  7. 2020Quantitative Propagation of Chaos for SGD in Wide Neural Networkscited by 20

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 1; 2023: 2; 2024: 4.

10 papers.

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2023Demystifying Structural Disparity in Graph Neural Networks: Can One Size Fit All?cited by 41
  3. 2022On Rademacher Complexity-based Generalization Bounds for Deep Learningcited by 16
  4. 2021Generalization Error Bounds on Deep Learning with Markov Datasetscited by 9
  5. 2021Conditionally Gaussian PAC-Bayescited by 7
  6. 2024Minimax optimality of deep neural networks on dependent data via PAC-Bayes boundscited by 4
  7. 2024On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspectivecited by 3
  8. 2024Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theorycited by 2
  9. 2023A note on regularised NTK dynamics with an application to PAC-Bayesian trainingcited by 1
  10. 2024Mixture of Scope Experts at Test: Generalizing Deeper Graph Neural Networks with Shallow Variantscited by 1