# Improving Infinitely Deep Bayesian Neural Networks with Nesterov's Accelerated Gradient Method

arXiv: https://arxiv.org/abs/2603.25024 · HTML: https://mlsanity.com/paper/2603.25024/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Neural Jump Stochastic Differential Equations](https://mlsanity.com/paper/1905.10403/) (2019)
- [Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints](https://mlsanity.com/paper/2010.13581/) (2020)
- [Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise](https://mlsanity.com/paper/1906.02355/) (2019)
- [Learning Long-Term Dependencies in Irregularly-Sampled Time Series](https://mlsanity.com/paper/2006.04418/) (2020)
- [On Second Order Behaviour in Augmented Neural ODEs](https://mlsanity.com/paper/2006.07220/) (2020)
- [Machine Learning from a Continuous Viewpoint](https://mlsanity.com/paper/1912.12777/) (2019)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations](https://mlsanity.com/paper/2102.06559/) (2021)
- [Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data](https://mlsanity.com/paper/2402.14989/) (2024)
- [Neural Ordinary Differential Equation Control of Dynamics on Graphs](https://mlsanity.com/paper/2006.09773/) (2020)
- [Comprehensive Review of Neural Differential Equations for Time Series Analysis](https://mlsanity.com/paper/2502.09885/) (2025)
- [Neural Ideal Large Eddy Simulation: Modeling Turbulence with Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.01174/) (2023)
- [Partially Stochastic Infinitely Deep Bayesian Neural Networks](https://mlsanity.com/paper/2402.03495/) (2024)

## Abstract (arXiv)

As a representative continuous-depth neural network approach, stochastic differential equation (SDE)-based Bayesian neural networks (BNNs) have attracted considerable attention due to their solid theoretical foundations and strong potential for real-world applications. However, their reliance on numerical SDE solvers inevitably incurs a large number of function evaluations (NFEs), resulting in high computational cost and occasional convergence instability. To address these challenges, we propose a Nesterov-accelerated gradient (NAG) enhanced SDE-BNN model. By integrating NAG into the SDE-BNN framework along with an NFE-dependent residual skip connection, our method accelerates convergence and substantially reduces NFEs during both training and testing. Extensive empirical results show that our model consistently outperforms conventional SDE-BNNs across various tasks, including image classification and sequence modeling, achieving lower NFEs and improved predictive accuracy.