# Disentangled Recurrent Wasserstein Autoencoder

arXiv: https://arxiv.org/abs/2101.07496 · HTML: https://mlsanity.com/paper/2101.07496/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulations](https://mlsanity.com/paper/1505.05116/) (2015)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [The Cramer Distance as a Solution to Biased Wasserstein Gradients](https://mlsanity.com/paper/1705.10743/) (2017)

## Built on by (most cited first, 22 of 22)

- [Improving Tuning-Free Real Image Editing with Proximal Guidance](https://mlsanity.com/paper/2306.05414/) (2023)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [Multifactor Sequential Disentanglement via Structured Koopman Autoencoders](https://mlsanity.com/paper/2303.17264/) (2023)
- [Temporally Disentangled Representation Learning under Unknown Nonstationarity](https://mlsanity.com/paper/2310.18615/) (2023)
- [Generalizing to Evolving Domains with Latent Structure-Aware Sequential Autoencoder](https://mlsanity.com/paper/2205.07649/) (2022)
- [Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation](https://mlsanity.com/paper/2305.15924/) (2023)
- [Combining audio control and style transfer using latent diffusion](https://mlsanity.com/paper/2408.00196/) (2024)
- [S3RP: Self-Supervised Super-Resolution and Prediction for Advection-Diffusion Process](https://mlsanity.com/paper/2111.04639/) (2021)
- [Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE](https://mlsanity.com/paper/2206.02512/) (2022)
- [Sequential Disentanglement by Extracting Static Information From A Single Sequence Element](https://mlsanity.com/paper/2406.18131/) (2024)
- [Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audio](https://mlsanity.com/paper/2205.05871/) (2022)
- [MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction](https://mlsanity.com/paper/2210.12418/) (2022)
- [Enhancing Evolving Domain Generalization through Dynamic Latent Representations](https://mlsanity.com/paper/2401.08464/) (2024)
- [Sequential Representation Learning via Static-Dynamic Conditional Disentanglement](https://mlsanity.com/paper/2408.05599/) (2024)
- [Generative Adversarial Network for Probabilistic Forecast of Random Dynamical System](https://mlsanity.com/paper/2111.03126/) (2021)
- [Unsupervised Representation Learning from Sparse Transformation Analysis](https://mlsanity.com/paper/2410.05564/) (2024)
- [Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video](https://mlsanity.com/paper/2509.08376/) (2025)
- [Learning Multimodal Latent Dynamics for Human-Robot Interaction](https://mlsanity.com/paper/2311.16380/) (2023)
- [DiViD: Disentangled Video Diffusion for Static-Dynamic Factorization](https://mlsanity.com/paper/2507.13934/) (2025)
- [Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations](https://mlsanity.com/paper/2510.17313/) (2025)
- [Factor Representation and Decision Making in Stock Markets Using Deep Reinforcement Learning](https://mlsanity.com/paper/2108.01758/) (2021)
- [Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2112.05343/) (2021)

## Abstract (arXiv)

Learning disentangled representations leads to interpretable models and facilitates data generation with style transfer, which has been extensively studied on static data such as images in an unsupervised learning framework. However, only a few works have explored unsupervised disentangled sequential representation learning due to challenges of generating sequential data. In this paper, we propose recurrent Wasserstein Autoencoder (R-WAE), a new framework for generative modeling of sequential data. R-WAE disentangles the representation of an input sequence into static and dynamic factors (i.e., time-invariant and time-varying parts). Our theoretical analysis shows that, R-WAE minimizes an upper bound of a penalized form of the Wasserstein distance between model distribution and sequential data distribution, and simultaneously maximizes the mutual information between input data and different disentangled latent factors, respectively. This is superior to (recurrent) VAE which does not explicitly enforce mutual information maximization between input data and disentangled latent representations. When the number of actions in sequential data is available as weak supervision information, R-WAE is extended to learn a categorical latent representation of actions to improve its disentanglement. Experiments on a variety of datasets show that our models outperform other baselines with the same settings in terms of disentanglement and unconditional video generation both quantitatively and qualitatively.