mlsanityData to 7 Oct 2026

Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge

arXiv 2005.09409 ↗First posted 19 May 2020eess.AS cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore vector quantization for acoustic unit discovery. Leveraging unlabelled data, we aim to learn discrete representations of speech that separate phonetic content from speaker-specific details. We propose two neural models to tackle this challenge - both use vector quantization to map continuous features to a finite set of codes. The first model is a type of vector-quantized variational autoencoder (VQ-VAE). The VQ-VAE encodes speech into a sequence of discrete units before reconstructing the audio waveform. Our second model combines vector quantization with contrastive predictive coding (VQ-CPC). The idea is to learn a representation of speech by predicting future acoustic units. We evaluate the models on English and Indonesian data for the ZeroSpeech 2020 challenge. In ABX phone discrimination tests, both models outperform all submissions to the 2019 and 2020 challenges, with a relative improvement of more than 30%. The models also perform competitively on a downstream voice conversion task. Of the two, VQ-CPC performs slightly better in general and is simpler and faster to train. Finally, probing experiments show that vector quantization is an effective bottleneck, forcing the models to discard speaker information.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 20; 2022: 18; 2023: 6; 2024: 5; 2025: 6.

55 papers.

Sort:

By area:

  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2021Unsupervised Speech Recognitioncited by 147
  5. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  6. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  7. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  8. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  9. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  10. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  11. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  12. 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
38 more
  1. 2023From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusioncited by 22
  2. 2021Aligned Contrastive Predictive Codingcited by 15
  3. 2021Adversarially learning disentangled speech representations for robust multi-factor voice conversioncited by 14
  4. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
  5. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  6. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  7. 2023Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignmentcited by 8
  8. 2021Double Articulation Analyzer with Prosody for Unsupervised Word and Phoneme Discoverycited by 7
  9. 2021Interpreting intermediate convolutional layers of generative CNNs trained on waveformscited by 6
  10. 2021Zero-shot Voice Conversion via Self-supervised Prosody Representation Learningcited by 6
  11. 2022VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversioncited by 6
  12. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  13. 2023Rhythm Modeling for Voice Conversioncited by 6
  14. 2021Unsupervised Automatic Speech Recognition: A Reviewcited by 5
  15. 2022Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech datacited by 5
  16. 2022Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAEcited by 5
  17. 2021A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robotscited by 4
  18. 2021NoiseVC: Towards High Quality Zero-Shot Voice Conversioncited by 4
  19. 2022Robust Vector Quantized-Variational Autoencodercited by 4
  20. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  21. 2025Towards Audio Token Compression in Large Audio Language Modelscited by 4
  22. 2021Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whalescited by 3
  23. 2021VQCPC-GAN: Variable-Length Adversarial Audio Synthesis Using Vector-Quantized Contrastive Predictive Codingcited by 3
  24. 2021Discrete representations in neural models of spoken languagecited by 3
  25. 2021Interpreting intermediate convolutional layers in unsupervised acoustic word classificationcited by 3
  26. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  27. 2021Unsupervised Acoustic Unit Discovery by Leveraging a Language-Independent Subword Discriminative Feature Representationcited by 2
  28. 2022Non-Parallel Voice Conversion for ASR Augmentationcited by 2
  29. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  30. 2024Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Modelscited by 2
  31. 2021Non-Intrusive Binaural Speech Intelligibility Prediction from Discrete Latent Representationscited by 1
  32. 2022Unsupervised Multimodal Word Discovery based on Double Articulation Analysis with Co-occurrence cuescited by 1
  33. 2022A Temporal Extension of Latent Dirichlet Allocation for Unsupervised Acoustic Unit Discoverycited by 1
  34. 2022Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Modelscited by 1
  35. 2023Representation Learning With Hidden Unit Clustering For Low Resource Speech Applicationscited by 1
  36. 2024Discrete Unit based Masking for Improving Disentanglement in Voice Conversioncited by 1
  37. 2022Modelling word learning and recognition using visually grounded speechcited by 0
  38. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0

50 most cited of 55 All 55 (JSON)