mlsanityData to 7 Oct 2026

Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks

arXiv 2307.07753 ↗First posted 15 Jul 2023cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a novel prior learning method for advancing generalization and uncertainty estimation in deep neural networks. The key idea is to exploit scalable and structured posteriors of neural networks as informative priors with generalization guarantees. Our learned priors provide expressive probabilistic representations at large scale, like Bayesian counterparts of pre-trained models on ImageNet, and further produce non-vacuous generalization bounds. We also extend this idea to a continual learning framework, where the favorable properties of our priors are desirable. Major enablers are our technical contributions: (1) the sums-of-Kronecker-product computations, and (2) the derivations and optimizations of tractable objectives that lead to improved generalization bounds. Empirically, we exhaustively show the effectiveness of this method for uncertainty estimation and generalization.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 45 of its 91 references that are in the map. First posted on arXiv 15 Jul 2023.

Builds on

45 of 91 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  9. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  12. 2014New insights and perspectives on the natural gradient methodcited by 435
33 more
  1. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 336
  4. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  5. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  6. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  7. 2015Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inferencecited by 159
  8. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  9. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  10. 2019A Primer on PAC-Bayesian Learningcited by 130
  11. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  12. 2021Bayesian Neural Network Priors Revisitedcited by 121
  13. 2020Tighter risk certificates for neural networkscited by 112
  14. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  15. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  16. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  17. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  18. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  19. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  20. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  21. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  22. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  23. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  24. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  25. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  26. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  27. 2018Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priorscited by 35
  28. 2019Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayescited by 32
  29. 2020Estimating Model Uncertainty of Neural Networks in Sparse Information Formcited by 31
  30. 2021Meta-Learning Reliable Priors in the Function Spacecited by 23
  31. 2019Bayesian Structure Adaptation for Continual Learningcited by 21
  32. 2021Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processescited by 12
  33. 2020Bayesian Optimization Meets Laplace Approximation for Robotic Introspectioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 11
  2. 2025General Uncertainty Estimation with Delta Variancescited by 3
  3. 2025The Bayesian Approach to Continual Learning: An Overviewcited by 1
  4. 2026SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertaintycited by 0