# Scaling A Simple Approach to Zero-Shot Speech Recognition

arXiv: https://arxiv.org/abs/2407.17852 · HTML: https://mlsanity.com/paper/2407.17852/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [wav2letter++: The Fastest Open-source Speech Recognition System](https://mlsanity.com/paper/1812.07625/) (2018)
- [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018)
- [ASR2K: Speech Recognition for Around 2000 Languages without Audio](https://mlsanity.com/paper/2209.02842/) (2022)
- [VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019](https://mlsanity.com/paper/1905.11449/) (2019)
- [Flashlight: Enabling Innovation in Tools for Machine Learning](https://mlsanity.com/paper/2201.12465/) (2022)
- [Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings](https://mlsanity.com/paper/1804.00316/) (2018)
- [AlloVera: A Multilingual Allophone Database](https://mlsanity.com/paper/2004.08031/) (2020)

## Built on by (most cited first, 8 of 8)

- [CS-FLEURS: A Massively Multilingual and Code-Switched Speech Dataset](https://mlsanity.com/paper/2509.14161/) (2025)
- [Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations](https://mlsanity.com/paper/2503.06273/) (2025)
- [SpeechTaxi: On Multilingual Semantic Speech Classification](https://mlsanity.com/paper/2409.06372/) (2024)
- [LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliteration](https://mlsanity.com/paper/2412.15299/) (2024)
- [Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking](https://mlsanity.com/paper/2409.18428/) (2024)
- [TalTech Systems for the Interspeech 2025 ML-SUPERB 2.0 Challenge](https://mlsanity.com/paper/2506.01458/) (2025)
- [MLMA: Towards Multilingual ASR With Mamba-based Architectures](https://mlsanity.com/paper/2510.18684/) (2025)
- [Kakugo: Distillation of Low-Resource Languages into Small Language Models](https://mlsanity.com/paper/2601.14051/) (2026)

## Abstract (arXiv)

Despite rapid progress in increasing the language coverage of automatic speech recognition, the field is still far from covering all languages with a known writing script. Recent work showed promising results with a zero-shot approach requiring only a small amount of text data, however, accuracy heavily depends on the quality of the used phonemizer which is often weak for unseen languages. In this paper, we present MMS Zero-shot a conceptually simpler approach based on romanization and an acoustic model trained on data in 1,078 different languages or three orders of magnitude more than prior art. MMS Zero-shot reduces the average character error rate by a relative 46% over 100 unseen languages compared to the best previous work. Moreover, the error rate of our approach is only 2.5x higher compared to in-domain supervised baselines, while our approach uses no labeled data for the evaluation languages at all.