mlsanityData to 7 Oct 2026

Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

arXiv 2303.01037 ↗First posted 2 Mar 2023cs.CL cs.SD eess.AScited by 228

Not on the map. Why?

Abstract (arXiv)

We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). Builds on 67 of its 89 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

67 of 89 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
38 more
  1. 2015Listen, Attend and Spellcited by 542
  2. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  3. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  4. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  6. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  9. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  10. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  11. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  12. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  13. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  14. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  15. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  16. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  17. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  18. 2020Unsupervised pretraining transfers well across languagescited by 100
  19. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  20. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  21. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  22. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  23. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  24. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  25. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  26. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  27. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  28. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  29. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  30. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  31. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  32. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  33. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  34. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  35. 2020Learning Robust and Multilingual Speech Representationscited by 39
  36. 2020Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memorycited by 35
  37. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  38. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 30

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 50; 2024: 104; 2025: 49; 2026: 25.

228 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  4. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  5. 2025Kimi-Audio Technical Reportcited by 210
  6. 2026Gemma 4 Technical Reportcited by 187
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  8. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  9. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  11. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  12. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
38 more
  1. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  2. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  3. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
  4. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  5. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  6. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  7. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 58
  8. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  9. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  10. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  11. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  12. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  13. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 32
  14. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  15. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  16. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  17. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  18. 2024ASR Error Correction using Large Language Modelscited by 23
  19. 2023Can Contextual Biasing Remain Effective with Whisper and GPT-2?cited by 21
  20. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  21. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  22. 2024Casablanca: Data and Models for Multidialectal Arabic Speech Recognitioncited by 20
  23. 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 19
  24. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  25. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  26. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  27. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  28. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  29. 2023LAraBench: Benchmarking Arabic AI with Large Language Modelscited by 16
  30. 2023Universal Automatic Phonetic Transcription into the International Phonetic Alphabetcited by 16
  31. 2023Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Modelscited by 16
  32. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  33. 2024DiarizationLM: Speaker Diarization Post-Processing with Large Language Modelscited by 16
  34. 2024Learn and Don't Forget: Adding a New Language to ASR Foundation Modelscited by 16
  35. 2024YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detectioncited by 16
  36. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 15
  37. 2024Towards Hierarchical Spoken Language Dysfluency Modelingcited by 15
  38. 2024HeAR -- Health Acoustic Representationscited by 15

50 most cited of 228 All 228 (JSON)