mlsanityData to 7 Oct 2026

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

arXiv 2111.09296 ↗First posted 17 Nov 2021cs.CL cs.SD eess.AScited by 589

Not on the map. Why?

Abstract (arXiv)

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.

Key facts. Cited by 589 papers in the map (data through 7 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

52 of 69 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2016Deep Networks with Stochastic Depthcited by 973
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  13. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  14. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  15. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  16. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  17. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  18. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  19. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  20. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  21. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  22. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  23. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  24. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  25. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  26. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  27. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  28. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  29. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  30. 2020Unsupervised pretraining transfers well across languagescited by 100
  31. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  32. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  33. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  34. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  35. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  36. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  37. 2020Learning Robust and Multilingual Speech Representationscited by 39
  38. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 70; 2023: 139; 2024: 139; 2025: 116; 2026: 124.

589 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  4. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  8. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  10. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  11. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  12. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
38 more
  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  2. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  4. 2023Modular Deep Learningcited by 82
  5. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  6. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  7. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  8. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  9. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  10. 2022Audio Self-supervised Learning: A Surveycited by 65
  11. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  12. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  13. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  14. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  15. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 51
  16. 2022ByT5 model for massively multilingual grapheme-to-phoneme conversioncited by 49
  17. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  18. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  19. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  20. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  21. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  22. 2024The VoicePrivacy 2024 Challenge Evaluation Plancited by 39
  23. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  24. 2023Transformers in Speech Processing: A Surveycited by 38
  25. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  26. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  27. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  28. 2023AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASRcited by 34
  29. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  30. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  31. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  32. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  33. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  34. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 30
  35. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  36. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  37. 2024XLSR-Mamba: A Dual-Column Bidirectional State Space Model for Spoofing Attack Detectioncited by 27
  38. 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26

50 most cited of 589 All 589 (JSON)