# Revisiting Structured Variational Autoencoders

arXiv: https://arxiv.org/abs/2305.16543 · HTML: https://mlsanity.com/paper/2305.16543/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Modeling Irregular Time Series with Continuous Recurrent Units](https://mlsanity.com/paper/2111.11344/) (2021)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Gaussian Process Prior Variational Autoencoders](https://mlsanity.com/paper/1810.11738/) (2018)

## Built on by (most cited first, 11 of 11)

- [eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modeling](https://mlsanity.com/paper/2403.01371/) (2024)
- [A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systems](https://mlsanity.com/paper/2509.21716/) (2025)
- [Maximum Likelihood Learning of Latent Dynamics Without Reconstruction](https://mlsanity.com/paper/2505.23569/) (2025)
- [Unbiased Learning of Deep Generative Models with Structured Discrete Representations](https://mlsanity.com/paper/2306.08230/) (2023)
- [AR-Flow VAE: A Structured Autoregressive Flow Prior Variational Autoencoder for Unsupervised Blind Source Separation](https://mlsanity.com/paper/2603.14441/) (2026)
- [StrEBM: A Structured Latent Energy-Based Model for Blind Source Separation](https://mlsanity.com/paper/2604.17381/) (2026)
- [The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling](https://mlsanity.com/paper/2605.31547/) (2026)
- [Integrative Variational Autoencoders for Generative Modeling of an Image Outcome with Multiple Input Images](https://mlsanity.com/paper/2402.02734/) (2024)
- [Learning Latent Spaces for Domain Generalization in Time Series Forecasting](https://mlsanity.com/paper/2412.11171/) (2024)
- [Efficient Learning of Deep State Space Models via Importance Smoothing](https://mlsanity.com/paper/2605.21108/) (2026)
- [RAMP: Recognition parametrisation by Amortised Message Passing](https://mlsanity.com/paper/2607.18883/) (2026)

## Abstract (arXiv)

Structured variational autoencoders (SVAEs) combine probabilistic graphical model priors on latent variables, deep neural networks to link latent variables to observed data, and structure-exploiting algorithms for approximate posterior inference. These models are particularly appealing for sequential data, where the prior can capture temporal dependencies. However, despite their conceptual elegance, SVAEs have proven difficult to implement, and more general approaches have been favored in practice. Here, we revisit SVAEs using modern machine learning tools and demonstrate their advantages over more general alternatives in terms of both accuracy and efficiency. First, we develop a modern implementation for hardware acceleration, parallelization, and automatic differentiation of the message passing algorithms at the core of the SVAE. Second, we show that by exploiting structure in the prior, the SVAE learns more accurate models and posterior distributions, which translate into improved performance on prediction tasks. Third, we show how the SVAE can naturally handle missing data, and we leverage this ability to develop a novel, self-supervised training approach. Altogether, these results show that the time is ripe to revisit structured variational autoencoders.