mlsanityData to 7 Oct 2026

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

arXiv 2006.11477 ↗First posted 20 Jun 2020cs.CL cs.LG cs.SDcited by 3,853

Not on the map. Why?

Cited by 114 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.

Key facts. Cited by 3,853 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 259; 2022: 581; 2023: 714; 2024: 794; 2025: 849; 2026: 656.

3,853 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
38 more
  1. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  6. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  8. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  9. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  10. 2022Multimodal Learning with Transformers: A Surveycited by 399
  11. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  13. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  14. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  15. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  16. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  17. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  18. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  19. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  20. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 311
  21. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  22. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  23. 2022AudioGen: Textually Guided Audio Generationcited by 292
  24. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  25. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  26. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  27. 2022Masked Autoencoders that Listencited by 271
  28. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  29. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  30. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  31. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  32. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  33. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  34. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  35. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  36. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  37. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  38. 2022cosFormer: Rethinking Softmax in Attentioncited by 212

50 most cited of 3,853 All 3,853 (JSON)