mlsanityData to 14 Oct 2026

vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

arXiv 1910.05453 ↗First posted 12 Oct 2019cs.CL cs.LGcited by 340

Not on the map. Why?

Abstract (arXiv)

We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.

Key facts. Cited by 340 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 85; 2022: 83; 2023: 71; 2024: 57; 2025: 29; 2026: 15.

340 papers.

Sort:

By area:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  8. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  9. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  11. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  12. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
38 more
  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  3. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  4. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  5. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  6. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  8. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  9. 2021Unsupervised Speech Recognitioncited by 148
  10. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  11. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  13. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  14. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  15. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  16. 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
  17. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  18. 2021Simple and Effective Zero-shot Cross-lingual Phoneme Recognitioncited by 79
  19. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  20. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  21. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  22. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  23. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  24. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  25. 2022Audio Self-supervised Learning: A Surveycited by 65
  26. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 64
  27. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  28. 2024Language Model Can Listen While Speakingcited by 63
  29. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  30. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  31. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  32. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  33. 2024Restructuring Vector Quantization with the Rotation Trickcited by 51
  34. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  35. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  36. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  37. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  38. 2023Regularized Vector Quantization for Tokenized Image Synthesiscited by 46

50 most cited of 340 All 340 (JSON)