# Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders

arXiv: https://arxiv.org/abs/2008.06892 · HTML: https://mlsanity.com/paper/2008.06892/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2020.

## Built on by (most cited first, 8 of 8)

- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders](https://mlsanity.com/paper/2102.06610/) (2021)
- [Interpreting intermediate convolutional layers of generative CNNs trained on waveforms](https://mlsanity.com/paper/2104.09489/) (2021)
- [Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion](https://mlsanity.com/paper/2311.08104/) (2023)
- [Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network](https://mlsanity.com/paper/2207.14568/) (2022)

## Abstract (arXiv)

Unsupervised representation learning of speech has been of keen interest in recent years, which is for example evident in the wide interest of the ZeroSpeech challenges. This work presents a new method for learning frame level representations based on WaveNet auto-encoders. Of particular interest in the ZeroSpeech Challenge 2019 were models with discrete latent variable such as the Vector Quantized Variational Auto-Encoder (VQVAE). However these models generate speech with relatively poor quality. In this work we aim to address this with two approaches: first WaveNet is used as the decoder and to generate waveform data directly from the latent representation; second, the low complexity of latent representations is improved with two alternative disentanglement learning methods, namely instance normalization and sliced vector quantization. The method was developed and tested in the context of the recent ZeroSpeech challenge 2020. The system output submitted to the challenge obtained the top position for naturalness (Mean Opinion Score 4.06), top position for intelligibility (Character Error Rate 0.15), and third position for the quality of the representation (ABX test score 12.5). These and further analysis in this paper illustrates that quality of the converted speech and the acoustic units representation can be well balanced.