# A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning

arXiv: https://arxiv.org/abs/1710.05741 · HTML: https://mlsanity.com/paper/1710.05741/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2017.

## Built on by (most cited first, 30 of 107)

- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models](https://mlsanity.com/paper/2208.09399/) (2022)
- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [Modeling Irregular Time Series with Continuous Recurrent Units](https://mlsanity.com/paper/2111.11344/) (2021)
- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Temporally Disentangled Representation Learning](https://mlsanity.com/paper/2210.13647/) (2022)
- [Criteria for Classifying Forecasting Methods](https://mlsanity.com/paper/2212.03523/) (2022)
- [ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models](https://mlsanity.com/paper/2106.10121/) (2021)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [Time Series Anomaly Detection for Cyber-Physical Systems via Neural System Identification and Bayesian Filtering](https://mlsanity.com/paper/2106.07992/) (2021)
- [Neural Continuous-Discrete State Space Models for Irregularly-Sampled Time Series](https://mlsanity.com/paper/2301.11308/) (2023)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [Auto-differentiable Ensemble Kalman Filters](https://mlsanity.com/paper/2107.07687/) (2021)
- [Multifactor Sequential Disentanglement via Structured Koopman Autoencoders](https://mlsanity.com/paper/2303.17264/) (2023)
- [DANSE: Data-driven Non-linear State Estimation of Model-free Process in Unsupervised Learning Setup](https://mlsanity.com/paper/2306.03897/) (2023)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Unsupervised Learned Kalman Filtering](https://mlsanity.com/paper/2110.09005/) (2021)
- [Deep Explicit Duration Switching Models for Time Series](https://mlsanity.com/paper/2110.13878/) (2021)
- [RNN with Particle Flow for Probabilistic Spatio-temporal Forecasting](https://mlsanity.com/paper/2106.06064/) (2021)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models](https://mlsanity.com/paper/2101.07046/) (2021)
- [Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations](https://mlsanity.com/paper/2201.08186/) (2022)
- [On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2210.09256/) (2022)
- [Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty](https://mlsanity.com/paper/2110.08607/) (2021)
- [KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter](https://mlsanity.com/paper/2407.13185/) (2024)
- [Efficient Dynamics Modeling in Interactive Environments with Koopman Theory](https://mlsanity.com/paper/2306.11941/) (2023)
- [Amortized Control of Continuous State Space Feynman-Kac Model for Irregular Time Series](https://mlsanity.com/paper/2410.05602/) (2024)
- [On the Identifiability of Switching Dynamical Systems](https://mlsanity.com/paper/2305.15925/) (2023)
- [KalMamba: Towards Efficient Probabilistic State Space Models for RL under Uncertainty](https://mlsanity.com/paper/2406.15131/) (2024)

## Abstract (arXiv)

This paper takes a step towards temporal reasoning in a dynamically changing video, not in the pixel space that constitutes its frames, but in a latent space that describes the non-linear dynamics of the objects in its world. We introduce the Kalman variational auto-encoder, a framework for unsupervised learning of sequential data that disentangles two latent representations: an object's representation, coming from a recognition model, and a latent state describing its dynamics. As a result, the evolution of the world can be imagined and missing data imputed, both without the need to generate high dimensional frames at each time step. The model is trained end-to-end on videos of a variety of simulated physical systems, and outperforms competing methods in generative and missing data imputation tasks.