mlsanityData to 7 Oct 2026

Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages

arXiv 2305.12606 ↗First posted 21 May 2023cs.CL cs.SD eess.AScited by 20

Not on the map. Why?

Abstract (arXiv)

Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each. However, there are thousands of spoken languages worldwide, and adapting to new languages is an important problem. In this work, we aim to understand which model adapts better to languages unseen during pre-training. We fine-tune both models on 13 unseen languages and 18 seen languages. Our results show that the number of hours seen per language and language family during pre-training is predictive of how the models compare, despite the significant differences in the pre-training methods.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

25 of 34 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  8. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  12. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
13 more
  1. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  2. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  3. 2020Unsupervised pretraining transfers well across languagescited by 100
  4. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 33
  5. 2019Massively Multilingual Adversarial Speech Recognitioncited by 28
  6. 2021Pseudo-Labeling for Massively Multilingual Speech Recognitioncited by 23
  7. 2021An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learningcited by 19
  8. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  9. 2022Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilitiescited by 16
  10. 2021Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0cited by 12
  11. 2022Adaptive multilingual speech recognition with pretrained modelscited by 9
  12. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  13. 2022C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrievalcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 4; 2026: 5.

20 papers.

Sort:

  1. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  2. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  3. 2023The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any languagecited by 14
  4. 2024What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluationscited by 12
  5. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  6. 2024Wav2Gloss: Generating Interlinear Glossed Text from Speechcited by 8
  7. 2024Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systemscited by 7
  8. 2024Multistage Fine-tuning Strategies for Automatic Speech Recognition in Low-resource Languagescited by 6
  9. 2024Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognitioncited by 5
  10. 2024Language Complexity and Speech Recognition Accuracy: Orthographic Complexity Hurts, Phonological Complexity Doesn'tcited by 4
  11. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  12. 2023Using fine-tuning and min lookahead beam search to improve Whispercited by 3
8 more
  1. 2025CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognitioncited by 1
  2. 2025How I Built ASR for Endangered Languages with a Spoken Dictionarycited by 1
  3. 2026Sequential Adapter Stacking for Cross-Lingual Low-Resource ASRcited by 1
  4. 2025Refining Transcripts With TV Subtitles by Prompt-Based Weakly Supervised Training of ASRcited by 0
  5. 2026Pretrained self-supervised speech models can recognize unseen consonantscited by 0
  6. 2026Evaluating the Effect of Linguistic Relatedness on Cross-Lingual Transfer in Large Multilingual Automatic Speech Recognitioncited by 0
  7. 2026A Speech Corpus for Mizo Automatic Speech Recognition: Whisper and SraVaani 1.0 Fine-Tuning with Morphology-Aware Evaluationcited by 0
  8. 2026The Hidden Cost of Digits: Number Normalization and WER in ASR Systemscited by 0