mlsanityData to 7 Oct 2026

Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training

arXiv 2104.01027 ↗First posted 2 Apr 2021cs.SD cs.CL cs.LGcited by 148

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning of speech representations has been a very active research area but most work is focused on a single domain such as read audio books for which there exist large quantities of labeled and unlabeled data. In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain. Our experiments show that using target domain data during pre-training leads to large performance improvements across a variety of setups. On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%. This has obvious practical implications since it is much easier to obtain unlabeled target domain data than labeled data. Moreover, we find that pre-training on multiple domains improves generalization performance on domains not seen during training. Code and models will be made available at https://github.com/pytorch/fairseq.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 2 Apr 2021.

Builds on

28 of 46 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2016Layer Normalizationcited by 5,579
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
16 more
  1. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  2. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  4. 2020Unsupervised pretraining transfers well across languagescited by 100
  5. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  6. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  7. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  8. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  9. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  10. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42
  11. 2020Learning Robust and Multilingual Speech Representationscited by 39
  12. 2020Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Trainingcited by 38
  13. 2020The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugmentcited by 19
  14. 2020An investigation of phone-based subword units for end-to-end speech recognitioncited by 12
  15. 2018Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognitioncited by 5
  16. 2018A Study of Enhancement, Augmentation, and Autoencoder Methods for Domain Adaptation in Distant Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 38; 2023: 44; 2024: 23; 2025: 10; 2026: 11.

148 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  5. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  6. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  8. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  9. 2021Unsupervised Speech Recognitioncited by 147
  10. 2022Generative Spoken Dialogue Language Modelingcited by 125
  11. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  12. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
38 more
  1. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  2. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  3. 2022Audio Self-supervised Learning: A Surveycited by 65
  4. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  5. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 52
  6. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  7. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 47
  8. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  9. 2023The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASRcited by 36
  10. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  11. 2022Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learningcited by 29
  12. 2024Masked Image Modeling: A Surveycited by 28
  13. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  14. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  15. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  16. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  17. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 22
  18. 2021Improved Language Identification Through Cross-Lingual Self-Supervised Learningcited by 20
  19. 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
  20. 2022Proficiency assessment of L2 spoken English using wav2vec 2.0cited by 20
  21. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 20
  22. 2023Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretrainingcited by 18
  23. 2022Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptationcited by 16
  24. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 15
  25. 2023Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognitioncited by 14
  26. 2021DeToxy: A Large-Scale Multimodal Dataset for Toxicity Classification in Spoken Utterancescited by 13
  27. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  28. 2021Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0cited by 12
  29. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  30. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  31. 2023Investigating Pre-trained Audio Encoders in the Low-Resource Conditioncited by 11
  32. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  33. 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 10
  34. 2022Boosting Cross-Domain Speech Recognition with Self-Supervisioncited by 10
  35. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10
  36. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  37. 2022DDOS: A MOS Prediction Framework utilizing Domain Adaptive Pre-training and Distribution of Opinion Scorescited by 7
  38. 2022Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greekcited by 7

50 most cited of 148 All 148 (JSON)