# Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models

arXiv: https://arxiv.org/abs/2101.07046 · HTML: https://mlsanity.com/paper/2101.07046/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 67 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Traffic Graph Convolutional Recurrent Neural Network: A Deep Learning Framework for Network-Scale Traffic Learning and Forecasting](https://mlsanity.com/paper/1802.07007/) (2018)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks](https://mlsanity.com/paper/1701.04722/) (2017)
- [A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning](https://mlsanity.com/paper/1710.05741/) (2017)
- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [Filtering Variational Objectives](https://mlsanity.com/paper/1705.09279/) (2017)
- [Variational inference for Monte Carlo objectives](https://mlsanity.com/paper/1602.06725/) (2016)
- [Variational Autoencoder with Arbitrary Conditioning](https://mlsanity.com/paper/1806.02382/) (2018)
- [Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction](https://mlsanity.com/paper/1801.02143/) (2018)
- [Learning Stochastic Recurrent Networks](https://mlsanity.com/paper/1411.7610/) (2014)
- [Variational Recurrent Auto-Encoders](https://mlsanity.com/paper/1412.6581/) (2014)
- [Learning Hierarchical Priors in VAEs](https://mlsanity.com/paper/1905.04982/) (2019)
- [Temporal Difference Variational Auto-Encoder](https://mlsanity.com/paper/1806.03107/) (2018)

## Built on by (most cited first, 14 of 14)

- [Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion](https://mlsanity.com/paper/2202.04770/) (2022)
- [TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning](https://mlsanity.com/paper/2312.15709/) (2023)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics](https://mlsanity.com/paper/2212.07892/) (2022)
- [Bounding Evidence and Estimating Log-Likelihood in VAE](https://mlsanity.com/paper/2206.09453/) (2022)
- [Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systems](https://mlsanity.com/paper/2305.01773/) (2023)
- [Laser: Latent Set Representations for 3D Generative Modeling](https://mlsanity.com/paper/2301.05747/) (2023)
- [Sampling-Free Probabilistic Deep State-Space Models](https://mlsanity.com/paper/2309.08256/) (2023)
- [Latent Matters: Learning Deep State-Space Models](https://mlsanity.com/paper/2602.23050/) (2026)
- [Benchmarking Generative Latent Variable Models for Speech](https://mlsanity.com/paper/2202.12707/) (2022)
- [On the Natural Gradient of the Evidence Lower Bound](https://mlsanity.com/paper/2307.11249/) (2023)
- [The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling](https://mlsanity.com/paper/2605.31547/) (2026)
- [Linear Variational State-Space Filtering](https://mlsanity.com/paper/2201.01353/) (2022)
- [Amortized backward variational inference in nonlinear state-space models](https://mlsanity.com/paper/2206.00319/) (2022)

## Abstract (arXiv)

Amortised inference enables scalable learning of sequential latent-variable models (LVMs) with the evidence lower bound (ELBO). In this setting, variational posteriors are often only partially conditioned. While the true posteriors depend, e.g., on the entire sequence of observations, approximate posteriors are only informed by past observations. This mimics the Bayesian filter -- a mixture of smoothing posteriors. Yet, we show that the ELBO objective forces partially-conditioned amortised posteriors to approximate products of smoothing posteriors instead. Consequently, the learned generative model is compromised. We demonstrate these theoretical findings in three scenarios: traffic flow, handwritten digits, and aerial vehicle dynamics. Using fully-conditioned approximate posteriors, performance improves in terms of generative modelling and multi-step prediction.