mlsanityData to 14 Oct 2026

Amortising Inference and Meta-Learning Priors in Neural Networks

arXiv 2602.08782 ↗First posted 9 Feb 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

One of the core facets of Bayesianism is in the updating of prior beliefs in light of new evidence -- so how can we maintain a Bayesian approach if we have no prior beliefs in the first place? This is one of the central challenges in the field of Bayesian deep learning, where it is not clear how to represent beliefs about a prediction task by prior distributions over model parameters. Bridging the fields of Bayesian deep learning and probabilistic meta-learning, we introduce a way to textitlearn a weights prior from a collection of datasets by introducing a way to perform per-dataset amortised variational inference. The model we develop can be viewed as a neural process whose latent variable is the set of weights of a BNN and whose decoder is the neural network parameterised by a sample of the latent variable itself. This unique model allows us to study the behaviour of Bayesian neural networks under well-specified priors, use Bayesian neural networks as flexible generative models, and perform desirable but previously elusive feats in neural processes such as within-task minibatching or meta-learning under extreme data-starvation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 80 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

45 of 80 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  5. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  6. 2018Conditional Neural Processescited by 426
  7. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  8. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  9. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  10. 2019Attentive Neural Processescited by 283
  11. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  12. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
33 more
  1. 2019Functional Variational Bayesian Neural Networkscited by 127
  2. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  3. 2021Bayesian Neural Network Priors Revisitedcited by 121
  4. 2018Meta-Learning Probabilistic Inference For Predictioncited by 110
  5. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  6. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  7. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  8. 2020Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processescited by 76
  9. 2019Sequential Neural Processescited by 55
  10. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  11. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  12. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  13. 2020Bootstrapping Neural Processescited by 40
  14. 2023Autoregressive Conditional Neural Processescited by 33
  15. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 32
  16. 2025Can Transformers Learn Full Bayesian Inference in Context?cited by 31
  17. 2022Latent Bottlenecked Attentive Neural Processescited by 26
  18. 2021Meta-Learning Reliable Priors in the Function Spacecited by 23
  19. 2024Amortized Probabilistic Conditioning for Optimization, Simulation and Inferencecited by 23
  20. 2024A Meta-Learning Approach to Bayesian Causal Discoverycited by 22
  21. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 20
  22. 2019Meta-Learning Mean Functions for Gaussian Processescited by 17
  23. 2022Environmental Sensor Placement with Convolutional Gaussian Neural Processescited by 14
  24. 2020Hierarchical Gaussian Process Priors for Bayesian Neural Network Weightscited by 13
  25. 2024Translation Equivariant Transformer Neural Processescited by 13
  26. 2024FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learningcited by 11
  27. 2022Convolutional Neural Processes for Inpainting Satellite Imagescited by 4
  28. 2024In-Context In-Context Learning with Transformer Neural Processescited by 4
  29. 2023Amortised Inference in Neural Networks for Small-Scale Probabilistic Meta-Learningcited by 3
  30. 2020A Neural Process Approach for Probabilistic Reconstruction of No-Data Gaps in Lunar Digital Elevation Mapscited by 2
  31. 2025Sparse Gaussian Neural Processescited by 2
  32. 2024BALI: Learning Neural Networks via Bayesian Layerwise Inferencecited by 1
  33. 2025Last Layer Empirical Bayescited by 1

Built on by

Not cited yet.