mlsanityData to 14 Oct 2026

Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning

arXiv 2309.15317 ↗First posted 26 Sep 2023cs.CL cs.AI cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

Multilingual self-supervised learning (SSL) has often lagged behind state-of-the-art (SOTA) methods due to the expenses and complexity required to handle many languages. This further harms the reproducibility of SSL, which is already limited to few research groups due to its resource usage. We show that more powerful techniques can actually lead to more efficient pre-training, opening SSL to more research groups. We propose WavLabLM, which extends WavLM's joint prediction and denoising to 40k hours of data across 136 languages. To build WavLabLM, we devise a novel multi-stage pre-training method, designed to address the language imbalance of multilingual data. WavLabLM achieves comparable performance to XLS-R on ML-SUPERB with less than 10% of the training data, making SSL realizable with academic compute. We show that further efficiency can be achieved with a vanilla HuBERT Base model, which can maintain 94% of XLS-R's performance with only 3% of the data, 4 GPUs, and limited trials. We open-source all code and models in ESPnet.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 44 of its 50 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

44 of 50 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
32 more
  1. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  7. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  11. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  13. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  14. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  15. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  16. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 133
  17. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  18. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  19. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  20. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  21. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  22. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  23. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  24. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  25. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  26. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  27. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  28. 2023Zambezi Voice: A Multilingual Speech Corpus for Zambian Languagescited by 18
  29. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  30. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  31. 2023Exploration on HuBERT with Multiple Resolutionscited by 7
  32. 2023Cross-Lingual Cross-Age Group Adaptation for Low-Resource Elderly Speech Emotion Recognitioncited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5; 2025: 2.

8 papers.

  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  3. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 27
  4. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 18
  5. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  6. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  7. 2024SpeechTaxi: On Multilingual Semantic Speech Classificationcited by 6
  8. 2025Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTCcited by 1