# mHuBERT-147: A Compact Multilingual HuBERT Model

arXiv: https://arxiv.org/abs/2406.06371 · HTML: https://mlsanity.com/paper/2406.06371/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 30 of 43)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation](https://mlsanity.com/paper/2305.16107/) (2023)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [THCHS-30 : A Free Chinese Speech Corpus](https://mlsanity.com/paper/1512.01882/) (2015)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations](https://mlsanity.com/paper/2211.04508/) (2022)
- [BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpus](https://mlsanity.com/paper/2207.03546/) (2022)

## Built on by (most cited first, 30 of 66)

- [UniVerse-1: Unified Audio-Video Generation via Stitching of Experts](https://mlsanity.com/paper/2509.06155/) (2025)
- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [AfriHuBERT: A self-supervised speech representation model for African languages](https://mlsanity.com/paper/2409.20201/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching](https://mlsanity.com/paper/2510.22950/) (2025)
- [DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models](https://mlsanity.com/paper/2410.24177/) (2024)
- [From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM](https://mlsanity.com/paper/2503.10620/) (2025)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings](https://mlsanity.com/paper/2509.15001/) (2025)
- [SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition](https://mlsanity.com/paper/2409.10429/) (2024)
- [Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding](https://mlsanity.com/paper/2501.06117/) (2025)
- [Voice of a Continent: Mapping Africa's Speech Technology Frontier](https://mlsanity.com/paper/2505.18436/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [SpeechMapper: Speech-to-text Embedding Projector for LLMs](https://mlsanity.com/paper/2601.20417/) (2026)
- [Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR](https://mlsanity.com/paper/2603.06164/) (2026)
- [Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech data](https://mlsanity.com/paper/2506.01439/) (2025)
- [LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space](https://mlsanity.com/paper/2603.29339/) (2026)
- [Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations](https://mlsanity.com/paper/2604.02102/) (2026)
- [Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversion](https://mlsanity.com/paper/2506.04013/) (2025)
- [SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition](https://mlsanity.com/paper/2506.22143/) (2025)
- [Speech Language Models for Under-Represented Languages: Insights from Wolof](https://mlsanity.com/paper/2509.15362/) (2025)
- [DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment](https://mlsanity.com/paper/2509.18987/) (2025)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [The Voice of Equity: A Systematic Evaluation of Bias Mitigation Techniques for Speech-Based Cognitive Impairment Detection Across Architectures and Demographics](https://mlsanity.com/paper/2601.16989/) (2026)
- [Sylber 2.0: A Universal Syllable Embedding](https://mlsanity.com/paper/2601.22306/) (2026)
- [Linguistically Informed Evaluation of Multilingual ASR for African Languages](https://mlsanity.com/paper/2602.04716/) (2026)
- [Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration](https://mlsanity.com/paper/2505.04457/) (2025)
- [Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC](https://mlsanity.com/paper/2505.24200/) (2025)

## Abstract (arXiv)

We present mHuBERT-147, the first general-purpose massively multilingual HuBERT speech representation model trained on 90K hours of clean, open-license data. To scale up the multi-iteration HuBERT approach, we use faiss-based clustering, achieving 5.2x faster label assignment than the original method. We also apply a new multilingual batching up-sampling strategy, leveraging both language and dataset diversity. After 3 training iterations, our compact 95M parameter mHuBERT-147 outperforms larger models trained on substantially more data. We rank second and first on the ML-SUPERB 10min and 1h leaderboards, with SOTA scores for 3 tasks. Across ASR/LID tasks, our model consistently surpasses XLS-R (300M params; 436K hours) and demonstrates strong competitiveness against the much larger MMS (1B params; 491K hours). Our findings indicate that mHuBERT-147 is a promising model for multilingual speech tasks, offering an unprecedented balance between high performance and parameter efficiency.