# Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders

arXiv: https://arxiv.org/abs/1906.08152 · HTML: https://mlsanity.com/paper/1906.08152/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 21 of 21)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioning](https://mlsanity.com/paper/2211.05385/) (2022)
- [Is Disentanglement enough? On Latent Representations for Controllable Music Generation](https://mlsanity.com/paper/2108.01450/) (2021)
- [Timbre transfer using image-to-image denoising diffusion implicit models](https://mlsanity.com/paper/2307.04586/) (2023)
- [Can3Tok: Canonical 3D Tokenization and Latent Modeling of Scene-Level 3D Gaussians](https://mlsanity.com/paper/2508.01464/) (2025)
- [Learning long-term music representations via hierarchical contextual constraints](https://mlsanity.com/paper/2202.06180/) (2022)
- [Towards Controllable Audio Texture Morphing](https://mlsanity.com/paper/2304.11648/) (2023)
- [DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation](https://mlsanity.com/paper/2408.10807/) (2024)
- [SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model](https://mlsanity.com/paper/2410.02144/) (2024)
- [WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion](https://mlsanity.com/paper/2409.15321/) (2024)
- [FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search](https://mlsanity.com/paper/2510.21667/) (2025)
- [DeepA: A Deep Neural Analyzer For Speech And Singing Vocoding](https://mlsanity.com/paper/2110.06434/) (2021)
- [CAESynth: Real-Time Timbre Interpolation and Pitch Control with Conditional Autoencoders](https://mlsanity.com/paper/2111.05174/) (2021)
- [HyperGANStrument: Instrument Sound Synthesis and Editing with Pitch-Invariant Hypernetworks](https://mlsanity.com/paper/2401.04558/) (2024)
- [The Interpretation Gap in Text-to-Music Generation Models](https://mlsanity.com/paper/2407.10328/) (2024)
- [Evaluating Disentangled Representations for Controllable Music Generation](https://mlsanity.com/paper/2602.10058/) (2026)
- [Timbre Classification of Musical Instruments with a Deep Learning Multi-Head Attention-Based Model](https://mlsanity.com/paper/2107.06231/) (2021)
- [LDC-VAE: A Latent Distribution Consistency Approach to Variational AutoEncoders](https://mlsanity.com/paper/2109.10640/) (2021)
- [Towards Cross-Cultural Analysis using Music Information Dynamics](https://mlsanity.com/paper/2111.12588/) (2021)
- [On the Geometry of Music Bandwidth Extension in Latent Spaces of Audio Codecs](https://mlsanity.com/paper/2608.03721/) (2026)

## Abstract (arXiv)

In this paper, we learn disentangled representations of timbre and pitch for musical instrument sounds. We adapt a framework based on variational autoencoders with Gaussian mixture latent distributions. Specifically, we use two separate encoders to learn distinct latent spaces for timbre and pitch, which form Gaussian mixture components representing instrument identity and pitch, respectively. For reconstruction, latent variables of timbre and pitch are sampled from corresponding mixture components, and are concatenated as the input to a decoder. We show the model efficacy by latent space visualization, and a quantitative analysis indicates the discriminability of these spaces, even with a limited number of instrument labels for training. The model allows for controllable synthesis of selected instrument sounds by sampling from the latent spaces. To evaluate this, we trained instrument and pitch classifiers using original labeled data. These classifiers achieve high accuracy when tested on our synthesized sounds, which verifies the model performance of controllable realistic timbre and pitch synthesis. Our model also enables timbre transfer between multiple instruments, with a single autoencoder architecture, which is evaluated by measuring the shift in posterior of instrument classification. Our in depth evaluation confirms the model ability to successfully disentangle timbre and pitch.