# Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

arXiv: https://arxiv.org/abs/2102.06559 · HTML: https://mlsanity.com/paper/2102.06559/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 60 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Latent ODEs for Irregularly-Sampled Time Series](https://mlsanity.com/paper/1907.03907/) (2019)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise](https://mlsanity.com/paper/1906.02355/) (2019)
- [Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference](https://mlsanity.com/paper/1703.09194/) (2017)
- [Theoretical guarantees for sampling and inference in generative models with latent diffusions](https://mlsanity.com/paper/1903.01608/) (2019)
- [Noisy Natural Gradient as Variational Inference](https://mlsanity.com/paper/1712.02390/) (2017)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates](https://mlsanity.com/paper/2008.10546/) (2020)
- [Bayesian Hypernetworks](https://mlsanity.com/paper/1710.04759/) (2017)
- [Bayesian Dark Knowledge](https://mlsanity.com/paper/1506.04416/) (2015)
- [Bayesian Neural Ordinary Differential Equations](https://mlsanity.com/paper/2012.07244/) (2020)
- [ODE2VAE: Deep generative second order ODEs with Bayesian neural networks](https://mlsanity.com/paper/1905.10994/) (2019)

## Built on by (most cited first, 30 of 38)

- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Efficient and Accurate Gradients for Neural SDEs](https://mlsanity.com/paper/2105.13493/) (2021)
- [Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimization](https://mlsanity.com/paper/2310.02679/) (2023)
- [Bayesian Learning via Neural Schr"odinger-F"ollmer Flows](https://mlsanity.com/paper/2111.10510/) (2021)
- [Constructing Neural Network-Based Models for Simulating Dynamical Systems](https://mlsanity.com/paper/2111.01495/) (2021)
- [Bayesian Neural Controlled Differential Equations for Treatment Effect Estimation](https://mlsanity.com/paper/2310.17463/) (2023)
- [Predicting the impact of treatments over time with uncertainty aware neural differential equations](https://mlsanity.com/paper/2202.11987/) (2022)
- [Transport meets Variational Inference: Controlled Monte Carlo Diffusions](https://mlsanity.com/paper/2307.01050/) (2023)
- [Learning Stochastic Dynamics with Statistics-Informed Neural Network](https://mlsanity.com/paper/2202.12278/) (2022)
- [Thermodynamic AI and the fluctuation frontier](https://mlsanity.com/paper/2302.06584/) (2023)
- [Stateful ODE-Nets using Basis Function Expansions](https://mlsanity.com/paper/2106.10820/) (2021)
- [Neural Ideal Large Eddy Simulation: Modeling Turbulence with Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.01174/) (2023)
- [GoTube: Scalable Stochastic Verification of Continuous-Depth Models](https://mlsanity.com/paper/2107.08467/) (2021)
- [Variational Inference for Infinitely Deep Neural Networks](https://mlsanity.com/paper/2209.10091/) (2022)
- [How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.06335/) (2023)
- [Uncertainty Modeling in Graph Neural Networks via Stochastic Differential Equations](https://mlsanity.com/paper/2408.16115/) (2024)
- [Partially Stochastic Infinitely Deep Bayesian Neural Networks](https://mlsanity.com/paper/2402.03495/) (2024)
- [Ornstein-Uhlenbeck Adaptation as a Mechanism for Learning in Brains and Machines](https://mlsanity.com/paper/2410.13563/) (2024)
- [Learning nonlinear integral operators via Recurrent Neural Networks and its application in solving Integro-Differential Equations](https://mlsanity.com/paper/2310.09434/) (2023)
- [Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence](https://mlsanity.com/paper/2305.15557/) (2023)
- [Uncertainty Quantification on Graph Learning: A Survey](https://mlsanity.com/paper/2404.14642/) (2024)
- [Fully Bayesian Differential Gaussian Processes through Stochastic Differential Equations](https://mlsanity.com/paper/2408.06069/) (2024)
- [Faster Training of Neural ODEs Using Gauss-Legendre Quadrature](https://mlsanity.com/paper/2308.10644/) (2023)
- [Graph Neural Stochastic Differential Equations](https://mlsanity.com/paper/2308.12316/) (2023)
- [Uncertainty quantification of neural network models of evolving processes via Langevin sampling](https://mlsanity.com/paper/2504.14854/) (2025)
- [Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models](https://mlsanity.com/paper/2106.11609/) (2021)
- [Characteristic Neural Ordinary Differential Equations](https://mlsanity.com/paper/2111.13207/) (2021)
- [Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings](https://mlsanity.com/paper/2205.11989/) (2022)
- [HGAN-SDEs: Learning Neural Stochastic Differential Equations with Hermite-Guided Adversarial Training](https://mlsanity.com/paper/2512.20272/) (2025)
- [Noisy Learning for Neural ODEs Acts as a Robustness Locus Widening](https://mlsanity.com/paper/2206.08237/) (2022)

## Abstract (arXiv)

We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.