# Learning Latent Representations for Speech Generation and Transformation

arXiv: https://arxiv.org/abs/1704.04222 · HTML: https://mlsanity.com/paper/1704.04222/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2017.

## Built on by (most cited first, 27 of 27)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [OmniAudio: Generating Spatial Audio from 360-Degree Video](https://mlsanity.com/paper/2504.14906/) (2025)
- [Emphasis control for parallel neural TTS](https://mlsanity.com/paper/2110.03012/) (2021)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Gromov-Wasserstein Autoencoders](https://mlsanity.com/paper/2209.07007/) (2022)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)
- [Pathological voice adaptation with autoencoder-based voice conversion](https://mlsanity.com/paper/2106.08427/) (2021)
- [Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data](https://mlsanity.com/paper/2203.11476/) (2022)
- [Learning robust speech representation with an articulatory-regularized variational autoencoder](https://mlsanity.com/paper/2104.03204/) (2021)
- [A learned conditional prior for the VAE acoustic space of a TTS system](https://mlsanity.com/paper/2106.10229/) (2021)
- [A Sparsity-promoting Dictionary Model for Variational Autoencoders](https://mlsanity.com/paper/2203.15758/) (2022)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [An Attribute-Aligned Strategy for Learning Speech Representation](https://mlsanity.com/paper/2106.02810/) (2021)
- [Cross-Utterance Conditioned VAE for Speech Generation](https://mlsanity.com/paper/2309.04156/) (2023)
- [Interference Motion Removal for Doppler Radar Vital Sign Detection Using Variational Encoder-Decoder Neural Network](https://mlsanity.com/paper/2404.08298/) (2024)
- [Towards the Next Frontier in Speech Representation Learning Using Disentanglement](https://mlsanity.com/paper/2407.02543/) (2024)
- [Towards Cross-Cultural Analysis using Music Information Dynamics](https://mlsanity.com/paper/2111.12588/) (2021)
- [Improved far-field speech recognition using Joint Variational Autoencoder](https://mlsanity.com/paper/2204.11286/) (2022)
- [Learning Multivariate CDFs and Copulas using Tensor Factorization](https://mlsanity.com/paper/2210.07132/) (2022)
- [Learning Invariant Representation and Risk Minimized for Unsupervised Accent Domain Adaptation](https://mlsanity.com/paper/2210.08182/) (2022)
- [Privacy-Utility Balanced Voice De-Identification Using Adversarial Examples](https://mlsanity.com/paper/2211.05446/) (2022)
- [An investigation of the reconstruction capacity of stacked convolutional autoencoders for log-mel-spectrograms](https://mlsanity.com/paper/2301.07665/) (2023)

## Abstract (arXiv)

An ability to model a generative process and learn a latent representation for speech in an unsupervised fashion will be crucial to process vast quantities of unlabelled speech data. Recently, deep probabilistic generative models such as Variational Autoencoders (VAEs) have achieved tremendous success in modeling natural images. In this paper, we apply a convolutional VAE to model the generative process of natural speech. We derive latent space arithmetic operations to disentangle learned latent representations. We demonstrate the capability of our model to modify the phonetic content or the speaker identity for speech segments using the derived operations, without the need for parallel supervisory data.