# Optimizing Bayesian Recurrent Neural Networks on an FPGA-based Accelerator

arXiv: https://arxiv.org/abs/2106.06048 · HTML: https://mlsanity.com/paper/2106.06048/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA](https://mlsanity.com/paper/1612.00694/) (2016)
- [Bayesian Recurrent Neural Networks](https://mlsanity.com/paper/1704.02798/) (2017)
- [VIBNN: Hardware Acceleration of Bayesian Neural Networks](https://mlsanity.com/paper/1802.00822/) (2018)
- [Bayesian Recurrent Neural Network Models for Forecasting and Quantifying Uncertainty in Spatial-Temporal Data](https://mlsanity.com/paper/1711.00636/) (2017)
- [Fault Detection and Identification using Bayesian Recurrent Neural Networks](https://mlsanity.com/paper/1911.04386/) (2019)
- [FINN-L: Library Extensions and Design Trade-off Analysis for Variable Precision LSTM Networks on FPGAs](https://mlsanity.com/paper/1807.04093/) (2018)
- [Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization](https://mlsanity.com/paper/2005.03857/) (2020)
- [Recurrent Neural Networks Hardware Implementation on FPGA](https://mlsanity.com/paper/1511.05552/) (2015)
- [High-Performance FPGA-based Accelerator for Bayesian Neural Networks](https://mlsanity.com/paper/2105.09163/) (2021)
- [Unrolling Ternary Neural Networks](https://mlsanity.com/paper/1909.04509/) (2019)
- [Bayesian LSTMs in medicine](https://mlsanity.com/paper/1706.01242/) (2017)

## Built on by (most cited first, 3 of 3)

- [Enhancing Dropout-based Bayesian Neural Networks with Multi-Exit on FPGA](https://mlsanity.com/paper/2406.14593/) (2024)
- [Accelerating MRI Uncertainty Estimation with Mask-based Bayesian Neural Network](https://mlsanity.com/paper/2407.05521/) (2024)
- [Algorithm and Hardware Co-Design for Efficient Complex-Valued Uncertainty Estimation](https://mlsanity.com/paper/2604.19993/) (2026)

## Abstract (arXiv)

Neural networks have demonstrated their outstanding performance in a wide range of tasks. Specifically recurrent architectures based on long-short term memory (LSTM) cells have manifested excellent capability to model time dependencies in real-world data. However, standard recurrent architectures cannot estimate their uncertainty which is essential for safety-critical applications such as in medicine. In contrast, Bayesian recurrent neural networks (RNNs) are able to provide uncertainty estimation with improved accuracy. Nonetheless, Bayesian RNNs are computationally and memory demanding, which limits their practicality despite their advantages. To address this issue, we propose an FPGA-based hardware design to accelerate Bayesian LSTM-based RNNs. To further improve the overall algorithmic-hardware performance, a co-design framework is proposed to explore the most fitting algorithmic-hardware configurations for Bayesian RNNs. We conduct extensive experiments on healthcare applications to demonstrate the improvement of our design and the effectiveness of our framework. Compared with GPU implementation, our FPGA-based design can achieve up to 10 times speedup with nearly 106 times higher energy efficiency. To the best of our knowledge, this is the first work targeting acceleration of Bayesian RNNs on FPGAs.