mlsanityData to 7 Oct 2026

W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

arXiv 2108.06209 ↗First posted 7 Aug 2021cs.LG cs.SD eess.AScited by 340

Not on the map. Why?

Abstract (arXiv)

Motivated by the success of masked language modeling~(MLM) in pre-training natural language processing models, we propose w2v-BERT that explores MLM for self-supervised speech representation learning. w2v-BERT is a framework that combines contrastive learning and MLM, where the former trains the model to discretize input continuous speech signals into a finite set of discriminative speech tokens, and the latter trains the model to learn contextualized speech representations via solving a masked prediction task consuming the discretized tokens. In contrast to existing MLM-based speech pre-training frameworks such as HuBERT, which relies on an iterative re-clustering and re-training process, or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT can be optimized in an end-to-end fashion by solving the two self-supervised tasks~(the contrastive task and MLM) simultaneously. Our experiments show that w2v-BERT achieves competitive results compared to current state-of-the-art pre-trained models on the LibriSpeech benchmarks when using the Libri-Light~60k corpus as the unsupervised data. In particular, when compared to published models such as conformer-based wav2vec~2.0 and HuBERT, our model shows~5% to~10% relative WER reduction on the test-clean and test-other subsets. When applied to the Google's Voice Search traffic dataset, w2v-BERT outperforms our internal conformer-based wav2vec~2.0 by more than~30% relatively.

Key facts. Cited by 340 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 7 Aug 2021.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
18 more
  1. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  2. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  4. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  5. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  6. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  7. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  8. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  9. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  10. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  11. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  12. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  13. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  14. 2019SpecAugment on Large Scale Datasetscited by 60
  15. 2020Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstructioncited by 38
  16. 2019Lessons from Building Acoustic Models with a Million Hours of Speechcited by 27
  17. 2020Improved Speech Representations with Multi-Target Autoregressive Predictive Codingcited by 23
  18. 2020Representation Learning for Sequence Data with Deep Autoencoding Predictive Componentscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 55; 2023: 89; 2024: 91; 2025: 60; 2026: 36.

340 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  6. 2023MusicLM: Generating Music From Textcited by 442
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  8. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  9. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  10. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  11. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  12. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
38 more
  1. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  2. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  3. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  4. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  5. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  6. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  7. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  8. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  9. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  10. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  11. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  12. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  13. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  14. 2023Textually Pretrained Speech Language Modelscited by 99
  15. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  16. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  17. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  18. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  19. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  20. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  21. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  22. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  23. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  24. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  25. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  26. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  27. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  28. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  29. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  30. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  31. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  32. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  33. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 58
  34. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  35. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  36. 2023SingSong: Generating musical accompaniments from singingcited by 54
  37. 2023Efficient Neural Music Generationcited by 50
  38. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49

50 most cited of 340 All 340 (JSON)