mlsanityData to 7 Oct 2026

Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge

arXiv 2005.11676 ↗First posted 24 May 2020cs.CL cs.LG cs.SDcited by 21

Not on the map. Why?

Abstract (arXiv)

In this paper, we report our submitted system for the ZeroSpeech 2020 challenge on Track 2019. The main theme in this challenge is to build a speech synthesizer without any textual information or phonetic labels. In order to tackle those challenges, we build a system that must address two major components such as 1) given speech audio, extract subword units in an unsupervised way and 2) re-synthesize the audio from novel speakers. The system also needs to balance the codebook performance between the ABX error rate and the bitrate compression rate. Our main contribution here is we proposed Transformer-based VQ-VAE for unsupervised unit discovery and Transformer-based inverter for the speech synthesis given the extracted codebook. Additionally, we also explored several regularization methods to improve performance even further.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 9; 2023: 3; 2024: 4; 2025: 1.

21 papers.

Sort:

  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2023Textually Pretrained Speech Language Modelscited by 99
  5. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  6. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  7. 2023Disentanglement via Latent Quantizationcited by 40
  8. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  9. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  10. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modelingcited by 17
  11. 2024Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Expertscited by 9
  12. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
9 more
  1. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
  2. 2022Artificial intelligence approaches for materials-by-design of energetic materials: state-of-the-art, challenges, and future directionscited by 6
  3. 2022Learning an Artificial Language for Knowledge-Sharing in Multilingual Translationcited by 4
  4. 2021Discrete representations in neural models of spoken languagecited by 3
  5. 2021Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attentioncited by 2
  6. 2022Deep Reinforcement Learning with Vector Quantized Encodingcited by 2
  7. 2024Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraintscited by 2
  8. 2023Self-Organising Neural Discrete Representation Learning `a la Kohonencited by 0
  9. 2025GmSLM : Generative Marmoset Spoken Language Modelingcited by 0