# Disentangled Sequential Autoencoder

arXiv: https://arxiv.org/abs/1803.02991 · HTML: https://mlsanity.com/paper/1803.02991/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2018.

## Built on by (most cited first, 30 of 120)

- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://mlsanity.com/paper/2203.09043/) (2022)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)
- [Temporally Disentangled Representation Learning](https://mlsanity.com/paper/2210.13647/) (2022)
- [Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling](https://mlsanity.com/paper/2102.13156/) (2021)
- [Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning](https://mlsanity.com/paper/2305.18499/) (2023)
- [LEO: Generative Latent Image Animator for Human Video Synthesis](https://mlsanity.com/paper/2305.03989/) (2023)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting](https://mlsanity.com/paper/2307.08243/) (2023)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion](https://mlsanity.com/paper/2203.16705/) (2022)
- [Multifactor Sequential Disentanglement via Structured Koopman Autoencoders](https://mlsanity.com/paper/2303.17264/) (2023)
- [CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process](https://mlsanity.com/paper/2401.14535/) (2024)
- [Disentangled Recurrent Wasserstein Autoencoder](https://mlsanity.com/paper/2101.07496/) (2021)
- [Temporally Disentangled Representation Learning under Unknown Nonstationarity](https://mlsanity.com/paper/2310.18615/) (2023)
- [Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective](https://mlsanity.com/paper/2208.07365/) (2022)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Online Variational Filtering and Parameter Learning](https://mlsanity.com/paper/2110.13549/) (2021)
- [Generalizing to Evolving Domains with Latent Structure-Aware Sequential Autoencoder](https://mlsanity.com/paper/2205.07649/) (2022)
- [Learning to Refactor Action and Co-occurrence Features for Temporal Action Localization](https://mlsanity.com/paper/2206.11493/) (2022)
- [MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos](https://mlsanity.com/paper/2409.02638/) (2024)
- [Decoupling Local and Global Representations of Time Series](https://mlsanity.com/paper/2202.02262/) (2022)
- [HiT-DVAE: Human Motion Generation via Hierarchical Transformer Dynamical VAE](https://mlsanity.com/paper/2204.01565/) (2022)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data](https://mlsanity.com/paper/2410.04814/) (2024)
- [Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations](https://mlsanity.com/paper/2201.08186/) (2022)
- [Learning Latent Causal Dynamics](https://mlsanity.com/paper/2202.04828/) (2022)

## Abstract (arXiv)

We present a VAE architecture for encoding and generating high dimensional sequential data, such as video or audio. Our deep generative model learns a latent representation of the data which is split into a static and dynamic part, allowing us to approximately disentangle latent time-dependent features (dynamics) from features which are preserved over time (content). This architecture gives us partial control over generating content and dynamics by conditioning on either one of these sets of features. In our experiments on artificially generated cartoon video clips and voice recordings, we show that we can convert the content of a given sequence into another one by such content swapping. For audio, this allows us to convert a male speaker into a female speaker and vice versa, while for video we can separately manipulate shapes and dynamics. Furthermore, we give empirical evidence for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.