# Bayesian Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1704.02798 · HTML: https://mlsanity.com/paper/1704.02798/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 30 of 39)

- [Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://mlsanity.com/paper/2308.16175/) (2023)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Conformal Prediction with Temporal Quantile Adjustments](https://mlsanity.com/paper/2205.09940/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU](https://mlsanity.com/paper/2305.17473/) (2023)
- [When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting](https://mlsanity.com/paper/2106.03904/) (2021)
- [Error-quantified Conformal Inference for Time Series](https://mlsanity.com/paper/2502.00818/) (2025)
- [Automatic variational inference with cascading flows](https://mlsanity.com/paper/2102.04801/) (2021)
- [Conformal Prediction Intervals with Temporal Dependence](https://mlsanity.com/paper/2205.12940/) (2022)
- [Uncertainty-aware abstention in medical diagnosis based on medical texts](https://mlsanity.com/paper/2502.18050/) (2025)
- [BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark](https://mlsanity.com/paper/2107.08189/) (2021)
- [Neural Conformal Control for Time Series Forecasting](https://mlsanity.com/paper/2412.18144/) (2024)
- [Short-term bus travel time prediction for transfer synchronization with intelligent uncertainty handling](https://mlsanity.com/paper/2104.06819/) (2021)
- [Optimizing Bayesian Recurrent Neural Networks on an FPGA-based Accelerator](https://mlsanity.com/paper/2106.06048/) (2021)
- [Bayesian Neural Network Language Modeling for Speech Recognition](https://mlsanity.com/paper/2208.13259/) (2022)
- [Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement](https://mlsanity.com/paper/2505.01766/) (2025)
- [Probabilistic modeling of lake surface water temperature using a Bayesian spatio-temporal graph convolutional neural network](https://mlsanity.com/paper/2109.13235/) (2021)
- [Distribution-informed Online Conformal Prediction](https://mlsanity.com/paper/2512.07770/) (2025)
- [Global Hierarchical Neural Networks using Hierarchical Softmax](https://mlsanity.com/paper/2308.01210/) (2023)
- [Hierarchical Prompts for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2401.11544/) (2024)
- [Variational Learning ISTA](https://mlsanity.com/paper/2407.06646/) (2024)
- [Single- to multi-fidelity history-dependent learning with uncertainty quantification and disentanglement: application to data-driven constitutive modeling](https://mlsanity.com/paper/2507.13416/) (2025)
- [Bayesian Low-Rank Factorization for Robust Model Adaptation](https://mlsanity.com/paper/2510.18723/) (2025)
- [Uncertainty Quantification for Scientific Machine Learning using Sparse Variational Gaussian Process Kolmogorov-Arnold Networks (SVGP KAN)](https://mlsanity.com/paper/2512.05306/) (2025)
- [Long short-term relevance learning](https://mlsanity.com/paper/2106.12694/) (2021)
- [Uncertainty Set Prediction of Aggregated Wind Power Generation based on Bayesian LSTM and Spatio-Temporal Analysis](https://mlsanity.com/paper/2110.03358/) (2021)
- [Last layer state space model for representation learning and uncertainty quantification](https://mlsanity.com/paper/2307.01566/) (2023)
- [A Bayesian Programming Approach to Car-following Model Calibration and Validation using Limited Data](https://mlsanity.com/paper/2307.10437/) (2023)

## Abstract (arXiv)

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.