mlsanityData to 14 Oct 2026

A PAC-Bayes oracle inequality for sparse neural networks

arXiv 2204.12392 ↗First posted 26 Apr 2022math.ST stat.ML stat.THcited by 2

Not on the map. Why?

Abstract (arXiv)

We study the Gibbs posterior distribution for sparse deep neural nets in a nonparametric regression setting. The posterior can be accessed via Metropolis-adjusted Langevin algorithms. Using a mixture over uniform priors on sparse sets of network weights, we prove an oracle inequality which shows that the method adapts to the unknown regularity and hierarchical structure of the regression function. The estimator achieves the minimax-optimal rate of convergence (up to a logarithmic factor).

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

16 of 36 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  4. 2016Error bounds for approximations with deep ReLU networkscited by 574
  5. 2015Weight Uncertainty in Neural Networkscited by 554
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  7. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  8. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  9. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  10. 2019A Primer on PAC-Bayesian Learningcited by 132
  11. 2020Tighter risk certificates for neural networkscited by 112
  12. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
4 more
  1. 2018Posterior Concentration for Sparse Deep Learningcited by 50
  2. 2019Convergence Rates of Variational Inference in Sparse Deep Learningcited by 27
  3. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  4. 2021Structured Stochastic Gradient MCMCcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1.

2 papers.

  1. 2022Streaming PAC-Bayes Gaussian process regression with a performance guarantee for online decision makingcited by 0
  2. 2023The surrogate Gibbs-posterior of a corrected stochastic MALA: Towards uncertainty quantification for neural networkscited by 0