mlsanityData to 14 Oct 2026

Unsupervised Speech Recognition

arXiv 2105.11084 ↗First posted 24 May 2021cs.CL cs.SD eess.AScited by 148

Not on the map. Why?

Abstract (arXiv)

Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe. This paper describes wav2vec-U, short for wav2vec Unsupervised, a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment unlabeled audio and learn a mapping from these representations to phonemes via adversarial training. The right representations are key to the success of our method. Compared to the best previous unsupervised work, wav2vec-U reduces the phoneme error rate on the TIMIT benchmark from 26.1 to 11.3. On the larger English Librispeech benchmark, wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago. We also experiment on nine other languages, including low-resource languages such as Kyrgyz, Swahili and Tatar.

Key facts. Cited by 148 papers in the map (data through 14 Oct 2026). Builds on 73 of its 139 references that are in the map. First posted on arXiv 24 May 2021.

Builds on

73 of 139 references.

References · 73

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  11. 2017Billion-scale similarity search with GPUscited by 2,497
  12. 2017Improved Training of Wasserstein GANscited by 2,235
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  8. 2017Wasserstein GANcited by 639
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  10. 2017Word Translation Without Parallel Datacited by 487
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  13. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  14. 2015Attention-Based Models for Speech Recognitioncited by 401
  15. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  16. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  17. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  18. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  19. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  20. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  21. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  22. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  23. 2017Unsupervised Neural Machine Translationcited by 149
  24. 2016Dual Learning for Machine Translationcited by 148
  25. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  26. 2019Self-Training for End-to-End Speech Recognitioncited by 129
  27. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  28. 2020Unsupervised pretraining transfers well across languagescited by 101
  29. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  30. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  31. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  32. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 96
  33. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  34. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 83
  35. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  36. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  37. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  38. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 55; 2023: 38; 2024: 23; 2025: 12; 2026: 3.

148 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  6. 2023A Cookbook of Self-Supervised Learningcited by 250
  7. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  8. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  9. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  10. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2021Simple and Effective Zero-shot Cross-lingual Phoneme Recognitioncited by 79
38 more
  1. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 79
  2. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  3. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  6. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  7. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  8. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 45
  9. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  10. 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 31
  11. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  12. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  13. 2022Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0cited by 27
  14. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
  15. 2022ASR2K: Speech Recognition for Around 2000 Languages without Audiocited by 26
  16. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  17. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  18. 2021CLSRIL-23: Cross Lingual Speech Representations for Indic Languagescited by 22
  19. 2022XTREME-S: Evaluating Cross-lingual Speech Representationscited by 22
  20. 2022KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stutteringcited by 21
  21. 2022Not always about you: Prioritizing community needs when developing endangered language technologycited by 21
  22. 2022Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Modelscited by 21
  23. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  24. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  25. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  26. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  27. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  28. 2021NWT: Towards natural audio-to-video generation with representation learningcited by 14
  29. 2022Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognitioncited by 14
  30. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
  31. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  32. 2023Multi-Scale Attention for Audio Question Answeringcited by 13
  33. 2022Simple and Effective Unsupervised Speech Synthesiscited by 12
  34. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  35. 2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
  36. 2024SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answeringcited by 12
  37. 2021Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASRcited by 11
  38. 2022Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpuscited by 10

50 most cited of 148 All 148 (JSON)