mlsanityData to 7 Oct 2026

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

arXiv 2110.13900 ↗First posted 26 Oct 2021cs.CL cs.SD eess.AScited by 1,609

Not on the map. Why?

Cited by 98 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.

Key facts. Cited by 1,609 papers in the map (data through 7 Oct 2026). Builds on 69 of its 95 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

69 of 95 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2016Deep Networks with Stochastic Depthcited by 973
  11. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  12. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
38 more
  1. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  5. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  7. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  8. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  12. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  13. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  14. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  15. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  16. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  17. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  18. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  19. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  20. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  21. 2020Unsupervised pretraining transfers well across languagescited by 100
  22. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  23. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  24. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  25. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  26. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  27. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 86
  28. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  29. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  30. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  31. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  32. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  33. 2020Continuous speech separation: dataset and analysiscited by 73
  34. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  35. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  36. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 70
  37. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  38. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 69

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 124; 2023: 230; 2024: 368; 2025: 434; 2026: 450.

1,609 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2025Qwen2.5-Omni Technical Reportcited by 704
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  5. 2024Qwen2-Audio Technical Reportcited by 543
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  7. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  9. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  10. 2022AudioGen: Textually Guided Audio Generationcited by 292
  11. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  12. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
38 more
  1. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  2. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  3. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  4. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  5. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  6. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  7. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  8. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  9. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  10. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  11. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  12. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  13. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  14. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  15. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  16. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  17. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  18. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  19. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  20. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  21. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  22. 2026Qwen3-TTS Technical Reportcited by 106
  23. 2025Step-Audio 2 Technical Reportcited by 101
  24. 2023Joint Audio and Speech Understandingcited by 99
  25. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  26. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  27. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  28. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  29. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  30. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  31. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
  32. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  33. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  34. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  35. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  36. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  37. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  38. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70

50 most cited of 1,609 All 1,609 (JSON)