# Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audio

arXiv: https://arxiv.org/abs/2205.05871 · HTML: https://mlsanity.com/paper/2205.05871/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data](https://mlsanity.com/paper/1709.07902/) (2017)
- [Deep Learning of Representations: Looking Forward](https://mlsanity.com/paper/1305.0445/) (2013)
- [S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation](https://mlsanity.com/paper/2005.11437/) (2020)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [Deep Variational Canonical Correlation Analysis](https://mlsanity.com/paper/1610.03454/) (2016)
- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer](https://mlsanity.com/paper/1809.07600/) (2018)
- [Progressive Learning and Disentanglement of Hierarchical Representations](https://mlsanity.com/paper/2002.10549/) (2020)
- [Disentangled Recurrent Wasserstein Autoencoder](https://mlsanity.com/paper/2101.07496/) (2021)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [NestedVAE: Isolating Common Factors via Weak Supervision](https://mlsanity.com/paper/2002.11576/) (2020)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [dMelodies: A Music Dataset for Disentanglement Learning](https://mlsanity.com/paper/2007.15067/) (2020)
- [VDSM: Unsupervised Video Disentanglement with State-Space Modeling and Deep Mixtures of Experts](https://mlsanity.com/paper/2103.07292/) (2021)

## Built on by (most cited first, 6 of 6)

- [Combining audio control and style transfer using latent diffusion](https://mlsanity.com/paper/2408.00196/) (2024)
- [Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer](https://mlsanity.com/paper/2409.06096/) (2024)
- [Sequential Representation Learning via Static-Dynamic Conditional Disentanglement](https://mlsanity.com/paper/2408.05599/) (2024)
- [DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation](https://mlsanity.com/paper/2408.10807/) (2024)
- [Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints](https://mlsanity.com/paper/2407.03824/) (2024)
- [Evaluating Disentangled Representations for Controllable Music Generation](https://mlsanity.com/paper/2602.10058/) (2026)

## Abstract (arXiv)

Disentangled sequential autoencoders (DSAEs) represent a class of probabilistic graphical models that describes an observed sequence with dynamic latent variables and a static latent variable. The former encode information at a frame rate identical to the observation, while the latter globally governs the entire sequence. This introduces an inductive bias and facilitates unsupervised disentanglement of the underlying local and global factors. In this paper, we show that the vanilla DSAE suffers from being sensitive to the choice of model architecture and capacity of the dynamic latent variables, and is prone to collapse the static latent variable. As a countermeasure, we propose TS-DSAE, a two-stage training framework that first learns sequence-level prior distributions, which are subsequently employed to regularise the model and facilitate auxiliary objectives to promote disentanglement. The proposed framework is fully unsupervised and robust against the global factor collapse problem across a wide range of model configurations. It also avoids typical solutions such as adversarial training which usually involves laborious parameter tuning, and domain-specific data augmentation. We conduct quantitative and qualitative evaluations to demonstrate its robustness in terms of disentanglement on both artificial and real-world music audio datasets.