mlsanityData to 14 Oct 2026

OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning

arXiv 2506.00338 ↗First posted 31 May 2025cs.CL cs.SD eess.AScited by 27

Not on the map. Why?

Abstract (arXiv)

The Open Whisper-style Speech Models (OWSM) project has developed a series of fully open speech foundation models using academic-scale resources, but their training data remains insufficient. This work enhances OWSM by integrating YODAS, a large-scale web-crawled dataset with a Creative Commons license. However, incorporating YODAS is nontrivial due to its wild nature, which introduces challenges such as incorrect language labels and audio-text misalignments. To address this, we develop a scalable data-cleaning pipeline using public toolkits, yielding a dataset with 166,000 hours of speech across 75 languages. Our new series of OWSM v4 models, trained on this curated dataset alongside existing OWSM data, significantly outperform previous versions on multilingual benchmarks. Our models even match or surpass frontier industrial models like Whisper and MMS in multiple scenarios. We will publicly release the cleaned YODAS data, pre-trained models, and all associated scripts via the ESPnet toolkit.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

36 of 41 references.

References · 36

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  11. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
24 more
  1. 2016FastText.zip: Compressing text classification modelscited by 335
  2. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  3. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  4. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  5. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  6. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  7. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  8. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  9. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 80
  10. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  11. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 76
  12. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  13. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 60
  14. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  15. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  16. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  17. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  18. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 14
  19. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 14
  20. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11
  21. 2022Unsupervised Data Selection via Discrete Speech Representation for ASRcited by 10
  22. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
  23. 2024MSR-86K: An Evolving, Multilingual Corpus with 86,300 Hours of Transcribed Audio for Speech Recognition Researchcited by 9
  24. 2022Unsupervised data selection for Speech Recognition with contrastive loss ratioscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 19.

27 papers.

Sort:

By area:

  1. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 23
  2. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 15
  3. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  4. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  5. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 5
  6. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  7. 2026Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesiscited by 3
  8. 2025CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguesecited by 2
  9. 2026A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASRcited by 1
  10. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 1
  11. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMscited by 1
  12. 2025Geolocation-Aware Robust Spoken Language Identificationcited by 0
15 more
  1. 2026Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliterationcited by 0
  2. 2026CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASRcited by 0
  3. 2026GLoRIA: Gated Low-Rank Interpretable Adaptation for Dialectal ASRcited by 0
  4. 2026Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?cited by 0
  5. 2026GigaAM Multilingual: Foundation Model for Underrepresented Languagescited by 0
  6. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0
  7. 2026Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactionscited by 0
  8. 2026Merging the Knowledge of LLMs for Automatic Speech Recognitioncited by 0
  9. 2026The Hidden Cost of Digits: Number Normalization and WER in ASR Systemscited by 0
  10. 2026HaikuS2S: A Cascaded System For Responding In Versecited by 0
  11. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0
  12. 2026Asymmetric Classifier-Free Guidance for Target-Speaker ASRcited by 0
  13. 2026FuseAlign: Forced Alignment in the Wildcited by 0
  14. 2026FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRscited by 0
  15. 2026Code-Switching Spoken Language Identification as Multi-Label Set Predictioncited by 0