mlsanityData to 14 Oct 2026

Sparse Uncertainty Representation in Deep Learning with Inducing Weights

arXiv 2105.14594 ↗First posted 30 May 2021stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Bayesian neural networks and deep ensembles represent two modern paradigms of uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due to memory inefficiency issues, since they require parameter storage several times higher than their deterministic counterparts. To address this, we augment the weight matrix of each layer with a small number of inducing weights, thereby projecting the uncertainty quantification into such low dimensional spaces. We further extend Matheron's conditional Gaussian sampling rule to enable fast weight sampling, which enables our inference method to maintain reasonable run-time as compared with ensembles. Importantly, our approach achieves competitive performance to the state-of-the-art in prediction and uncertainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size to ≤ 24.3% of that of a single neural network.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 47 of its 87 references that are in the map. First posted on arXiv 30 May 2021.

Builds on

47 of 87 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2013Auto-Encoding Variational Bayescited by 4,053
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
35 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  4. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2015Weight Uncertainty in Neural Networkscited by 554
  7. 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
  8. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  9. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  10. 2017Advances in Variational Inferencecited by 236
  11. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  12. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  13. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  14. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  15. 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
  16. 2019Functional Variational Bayesian Neural Networkscited by 127
  17. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 127
  18. 2017Bayesian Compression for Deep Learningcited by 116
  19. 2019Subspace Inference for Bayesian Deep Learningcited by 95
  20. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  21. 2015Hierarchical Variational Modelscited by 89
  22. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  23. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 80
  24. 2020Pathwise Conditioning of Gaussian Processescited by 64
  25. 2018Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameterscited by 58
  26. 2017Model Selection in Bayesian Neural Networks via Horseshoe Priorscited by 52
  27. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  28. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  29. 2018Variational Implicit Processescited by 41
  30. 2018SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradientcited by 34
  31. 2020A Tutorial on Sparse Gaussian Processes and Variational Inferencecited by 31
  32. 2017Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolutioncited by 30
  33. 2019An Adaptive Empirical Bayesian Method for Sparse Deep Learningcited by 20
  34. 2020Hierarchical Gaussian Process Priors for Bayesian Neural Network Weightscited by 13
  35. 2018The Deep Weight Priorcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 0; 2023: 4; 2024: 4; 2025: 3; 2026: 2.

16 papers.

Sort:

  1. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 81
  2. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  3. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  4. 2023Calibrating Multimodal Learningcited by 17
  5. 2021Function-space Inference with Sparse Implicit Processescited by 13
  6. 2021Structured Stochastic Gradient MCMCcited by 12
  7. 2025Revisiting Kernel Attention with Correlated Gaussian Process Representationcited by 10
  8. 2024Training Bayesian Neural Networks with Sparse Subspace Variational Inferencecited by 7
  9. 2021TyXe: Pyro-based Bayesian neural nets for Pytorchcited by 6
  10. 2024Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processescited by 5
  11. 2023Density Uncertainty Layers for Reliable Uncertainty Estimationcited by 4
  12. 2024Sparse Variational Student-t Processes for Heavy-tailed Modelingcited by 2
4 more
  1. 2026Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Modelscited by 2
  2. 2025Bayesian Computation in Deep Learningcited by 1
  3. 2025Flow-Induced Diagonal Gaussian Processescited by 1
  4. 2026Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibrationcited by 0