# A Bayesian Approach to Recurrence in Neural Networks

arXiv: https://arxiv.org/abs/1910.11247 · HTML: https://mlsanity.com/paper/1910.11247/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Mitigating Prior Errors in Causal Structure Learning: A Resilient Approach via Bayesian Networks](https://mlsanity.com/paper/2306.07032/) (2023)
- [Bayesian Recurrent Units and the Forward-Backward Algorithm](https://mlsanity.com/paper/2207.10486/) (2022)
- [Feature Preserving Shrinkage on Bayesian Neural Networks via the R2D2 Prior](https://mlsanity.com/paper/2505.18280/) (2025)
- [Variational Polya Tree](https://mlsanity.com/paper/2510.22651/) (2025)

## Abstract (arXiv)

We begin by reiterating that common neural network activation functions have simple Bayesian origins. In this spirit, we go on to show that Bayes's theorem also implies a simple recurrence relation; this leads to a Bayesian recurrent unit with a prescribed feedback formulation. We show that introduction of a context indicator leads to a variable feedback that is similar to the forget mechanism in conventional recurrent units. A similar approach leads to a probabilistic input gate. The Bayesian formulation leads naturally to the two pass algorithm of the Kalman smoother or forward-backward algorithm, meaning that inference naturally depends upon future inputs as well as past ones. Experiments on speech recognition confirm that the resulting architecture can perform as well as a bidirectional recurrent network with the same number of parameters as a unidirectional one. Further, when configured explicitly bidirectionally, the architecture can exceed the performance of a conventional bidirectional recurrence.