# Handling Missing Observations with an RNN-based Prediction-Update Cycle

arXiv: https://arxiv.org/abs/2103.11747 · HTML: https://mlsanity.com/paper/2103.11747/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Modeling Missing Data in Clinical Time Series with RNNs](https://mlsanity.com/paper/1606.04130/) (2016)
- [Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs](https://mlsanity.com/paper/1904.09507/) (2019)
- [Deep Learning for Vision-based Prediction: A Survey](https://mlsanity.com/paper/2007.00095/) (2020)
- [MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses](https://mlsanity.com/paper/1805.00652/) (2018)
- [Convolutional Neural Network for Trajectory Prediction](https://mlsanity.com/paper/1809.00696/) (2018)
- [Pedestrian Trajectory Prediction using Context-Augmented Transformer Networks](https://mlsanity.com/paper/2012.01757/) (2020)
- [Microscopic Pedestrian Flow Characteristics: Development of an Image Processing Data Collection and Simulation Model](https://mlsanity.com/paper/1610.00029/) (2016)

## Built on by (most cited first, 2 of 2)

- [MissFormer: (In-)attention-based handling of missing observations for trajectory filtering and prediction](https://mlsanity.com/paper/2106.16009/) (2021)
- [Meta-Imputation Balanced (MIB): An Ensemble Approach for Handling Missing Data in Biomedical Machine Learning](https://mlsanity.com/paper/2509.03316/) (2025)

## Abstract (arXiv)

In tasks such as tracking, time-series data inevitably carry missing observations. While traditional tracking approaches can handle missing observations, recurrent neural networks (RNNs) are designed to receive input data in every step. Furthermore, current solutions for RNNs, like omitting the missing data or data imputation, are not sufficient to account for the resulting increased uncertainty. Towards this end, this paper introduces an RNN-based approach that provides a full temporal filtering cycle for motion state estimation. The Kalman filter inspired approach, enables to deal with missing observations and outliers. For providing a full temporal filtering cycle, a basic RNN is extended to take observations and the associated belief about its accuracy into account for updating the current state. An RNN prediction model, which generates a parametrized distribution to capture the predicted states, is combined with an RNN update model, which relies on the prediction model output and the current observation. By providing the model with masking information, binary-encoded missing events, the model can overcome limitations of standard techniques for dealing with missing input values. The model abilities are demonstrated on synthetic data reflecting prototypical pedestrian tracking scenarios.