mlsanityData to 7 Oct 2026

mHuBERT-147: A Compact Multilingual HuBERT Model

arXiv 2406.06371 ↗First posted 10 Jun 2024cs.CL cs.SD eess.AScited by 64

Not on the map. Why?

Abstract (arXiv)

We present mHuBERT-147, the first general-purpose massively multilingual HuBERT speech representation model trained on 90K hours of clean, open-license data. To scale up the multi-iteration HuBERT approach, we use faiss-based clustering, achieving 5.2x faster label assignment than the original method. We also apply a new multilingual batching up-sampling strategy, leveraging both language and dataset diversity. After 3 training iterations, our compact 95M parameter mHuBERT-147 outperforms larger models trained on substantially more data. We rank second and first on the ML-SUPERB 10min and 1h leaderboards, with SOTA scores for 3 tasks. Across ASR/LID tasks, our model consistently surpasses XLS-R (300M params; 436K hours) and demonstrates strong competitiveness against the much larger MMS (1B params; 491K hours). Our findings indicate that mHuBERT-147 is a promising model for multilingual speech tasks, offering an unprecedented balance between high performance and parameter efficiency.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

43 of 58 references.

References · 43

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
31 more
  1. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  2. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  3. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  4. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  5. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  6. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  7. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  8. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  9. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  10. 2020Unsupervised pretraining transfers well across languagescited by 100
  11. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  12. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  13. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  14. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 46
  15. 2015THCHS-30 : A Free Chinese Speech Corpuscited by 43
  16. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  17. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  18. 2022BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpuscited by 31
  19. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  20. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  21. 2019JVS corpus: free Japanese multi-speaker voice corpuscited by 20
  22. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  23. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  24. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 15
  25. 2020ClovaCall: Korean Goal-Oriented Dialog Speech Corpus for Automatic Speech Recognition of Contact Centerscited by 13
  26. 2022Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannadacited by 10
  27. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  28. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  29. 2021Kosp2e: Korean Speech to English Translation Corpuscited by 7
  30. 2022Knowledge-driven Subword Grammar Modeling for Automatic Speech Recognition in Tamil and Kannadacited by 6
  31. 2022Improved Speech Pre-Training with Supervision-Enhanced Acoustic Unitcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 26; 2026: 33.

64 papers.

Sort:

By area:

  1. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  2. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 54
  3. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  4. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  5. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matchingcited by 14
  6. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  7. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 7
  8. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 7
  9. 2024Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programmingcited by 6
  10. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  11. 2025BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordingscited by 5
  12. 2024SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognitioncited by 4
38 more
  1. 2025Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understandingcited by 4
  2. 2025Voice of a Continent: Mapping Africa's Speech Technology Frontiercited by 4
  3. 2025Unsupervised lexicon learning from speech is limited by representations rather than clusteringcited by 4
  4. 2026Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTORcited by 4
  5. 2025Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech datacited by 3
  6. 2026SpeechMapper: Speech-to-text Embedding Projector for LLMscited by 3
  7. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  8. 2026Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representationscited by 3
  9. 2025Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversioncited by 2
  10. 2025SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognitioncited by 2
  11. 2025Speech Language Models for Under-Represented Languages: Insights from Wolofcited by 2
  12. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  13. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  14. 2026The Voice of Equity: A Systematic Evaluation of Bias Mitigation Techniques for Speech-Based Cognitive Impairment Detection Across Architectures and Demographicscited by 2
  15. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  16. 2026Linguistically Informed Evaluation of Multilingual ASR for African Languagescited by 2
  17. 2025Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restorationcited by 1
  18. 2025Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTCcited by 1
  19. 2025Factorized RVQ-GAN For Disentangled Speech Tokenizationcited by 1
  20. 2025SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challengecited by 1
  21. 2025Towards few-shot isolated word reading assessmentcited by 1
  22. 2025National Institute on Aging PREPARE Challenge: Early Detection of Cognitive Impairment Using Speech -- The SpeechCARE Solutioncited by 1
  23. 2025How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfercited by 1
  24. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  25. 2026A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASRcited by 1
  26. 2026Quantifying Cross-Lingual Transfer in Paralinguistic Speech Taskscited by 1
  27. 2026EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agentscited by 1
  28. 2026Language Orthogonalization for Zero-Shot Cross-Lingual Audio Deepfake Detectioncited by 1
  29. 2024The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Languagecited by 0
  30. 2025Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classificationcited by 0
  31. 2026Lightweight and perceptually-guided voice conversion for electro-laryngeal speechcited by 0
  32. 2026MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgettingcited by 0
  33. 2026Windowed SummaryMixing: An Efficient Fine-Tuning of Self-Supervised Learning Models for Low-resource Speech Recognitioncited by 0
  34. 2026Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Textcited by 0
  35. 2026Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgettingcited by 0
  36. 2026Giving Voice to the Constitution: Low-Resource Text-to-Speech for Quechua and Spanish Using a Bilingual Legal Corpuscited by 0
  37. 2026UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinationscited by 0
  38. 2026Polyglot: Multilingual Style Preserving Speech-Driven Facial Animationcited by 0

50 most cited of 64 All 64 (JSON)