mlsanityData to 7 Oct 2026

On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models

arXiv 2406.09282 ↗First posted 13 Jun 2024cs.CL cs.SD eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

The Open Whisper-style Speech Model (OWSM) series was introduced to achieve full transparency in building advanced speech-to-text (S2T) foundation models. To this end, OWSM models are trained on 25 public speech datasets, which are heterogeneous in multiple ways. In this study, we advance the OWSM series by introducing OWSM v3.2, which improves on prior models by investigating and addressing the impacts of this data heterogeneity. Our study begins with a detailed analysis of each dataset, from which we derive two key strategies: data filtering with proxy task to enhance data quality, and the incorporation of punctuation and true-casing using an open large language model (LLM). With all other configurations staying the same, OWSM v3.2 improves performance over the OWSM v3.1 baseline while using 15% less training data.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

31 of 50 references.

References · 31

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2023Mistral 7Bcited by 3,065
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  6. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  8. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  9. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  11. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  12. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
19 more
  1. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  2. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  3. 2023End-to-End Speech Recognition: A Surveycited by 127
  4. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  5. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  6. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 77
  7. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  8. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  9. 2022Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Datasetcited by 36
  10. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  11. 2021NeMo Inverse Text Normalization: From Development To Productioncited by 28
  12. 2021JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verificationcited by 21
  13. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  14. 2023Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognitioncited by 11
  15. 2022Unsupervised Data Selection via Discrete Speech Representation for ASRcited by 10
  16. 2023Updated Corpora and Benchmarks for Long-Form Speech Recognitioncited by 10
  17. 2022Unsupervised data selection for Speech Recognition with contrastive loss ratioscited by 7
  18. 2022Improving Noisy Student Training on Non-target Domain Data for Automatic Speech Recognitioncited by 5
  19. 2022Improving Data Driven Inverse Text Normalization using Data Augmentationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  2. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  3. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  4. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  5. 2025OLMoASR: Open Models and Data for Training Robust Speech Recognition Modelscited by 4
  6. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  7. 2024Efficiently Identifying Low-Quality Language Subsets in Multilingual Datasets: A Case Study on a Large-Scale Multilingual Audio Datasetcited by 1
  8. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  9. 2026Adapting Text LLMs to Speech via Multimodal Depth Up-Scalingcited by 0
  10. 2026GigaAM Multilingual: Foundation Model for Underrepresented Languagescited by 0