# Recurrent Neural Networks and Universal Approximation of Bayesian Filters

arXiv: https://arxiv.org/abs/2211.00335 · HTML: https://mlsanity.com/paper/2211.00335/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 68 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces](https://mlsanity.com/paper/1905.07357/) (2019)
- [How to Construct Deep Recurrent Neural Networks](https://mlsanity.com/paper/1312.6026/) (2013)
- [Probabilistic Recurrent State-Space Models](https://mlsanity.com/paper/1801.10395/) (2018)
- [Backprop KF: Learning Discriminative Deterministic State Estimators](https://mlsanity.com/paper/1605.07148/) (2016)
- [Variational Gaussian Process State-Space Models](https://mlsanity.com/paper/1406.4905/) (2014)
- [Neural Adaptive Sequential Monte Carlo](https://mlsanity.com/paper/1506.03338/) (2015)
- [Variational Data Assimilation with a Learned Inverse Observation Operator](https://mlsanity.com/paper/2102.11192/) (2021)
- [Overcoming Mean-Field Approximations in Recurrent Gaussian Process Models](https://mlsanity.com/paper/1906.05828/) (2019)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [State Space Gaussian Processes with Non-Gaussian Likelihood](https://mlsanity.com/paper/1802.04846/) (2018)
- [Lipschitz neural networks are dense in the set of all Lipschitz functions](https://mlsanity.com/paper/2009.13881/) (2020)
- [Auxiliary Likelihood-Based Approximate Bayesian Computation in State Space Models](https://mlsanity.com/paper/1604.07949/) (2016)
- [On the Mathematical Theory of Ensemble (Linear-Gaussian) Kalman-Bucy Filtering](https://mlsanity.com/paper/2006.08843/) (2020)
- [On the Stability of Kalman-Bucy Diffusion Processes](https://mlsanity.com/paper/1610.04686/) (2016)

## Built on by (most cited first, 5 of 5)

- [Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis](https://mlsanity.com/paper/2402.12241/) (2024)
- [Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals](https://mlsanity.com/paper/2310.19603/) (2023)
- [Universal Time-Uniform Trajectory Approximation for Random Dynamical Systems with Recurrent Neural Networks](https://mlsanity.com/paper/2211.08018/) (2022)
- [Quickest Detection of Hallucination Onset: Delay Bounds and Learned CUSUM Statistics](https://mlsanity.com/paper/2606.12476/) (2026)
- [How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL](https://mlsanity.com/paper/2609.11132/) (2026)

## Abstract (arXiv)

We consider the Bayesian optimal filtering problem: i.e. estimating some conditional statistics of a latent time-series signal from an observation sequence. Classical approaches often rely on the use of assumed or estimated transition and observation models. Instead, we formulate a generic recurrent neural network framework and seek to learn directly a recursive mapping from observational inputs to the desired estimator statistics. The main focus of this article is the approximation capabilities of this framework. We provide approximation error bounds for filtering in general non-compact domains. We also consider strong time-uniform approximation error bounds that guarantee good long-time performance. We discuss and illustrate a number of practical concerns and implications of these results.