mlsanityData to 14 Oct 2026

Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data

arXiv 2309.13876 ↗First posted 25 Sep 2023cs.CL cs.SD eess.AScited by 42

Not on the map. Why?

Abstract (arXiv)

Pre-training speech models on large volumes of data has achieved remarkable success. OpenAI Whisper is a multilingual multitask model trained on 680k hours of supervised speech data. It generalizes well to various speech recognition and translation benchmarks even in a zero-shot setup. However, the full pipeline for developing such models (from data collection to training) is not publicly accessible, which makes it difficult for researchers to further improve its performance and address training-related issues such as efficiency, robustness, fairness, and bias. This work presents an Open Whisper-style Speech Model (OWSM), which reproduces Whisper-style training using an open-source toolkit and publicly available data. OWSM even supports more translation directions and can be more efficient to train. We will publicly release all scripts used for data preparation, training, inference, and scoring as well as pre-trained models and training logs to promote open science.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

52 of 70 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  12. 2021Dynamic Neural Networks: A Surveycited by 408
  13. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  14. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  15. 2022A Survey of Machine Unlearningcited by 312
  16. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  17. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  18. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  19. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  20. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  21. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  22. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  23. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  24. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  25. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  26. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  27. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  28. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  29. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  30. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  31. 2022Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Datasetcited by 40
  32. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  33. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  34. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  35. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  36. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  37. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 14
  38. 2022Structured State Space Decoder for Speech Recognition and Synthesiscited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 19; 2026: 8.

42 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  3. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 67
  4. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  5. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  6. 2024Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Modelscited by 29
  7. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 27
  8. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  9. 2025OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academiacited by 13
  10. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
  11. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
  12. 2025Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Usecited by 9
30 more
  1. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  2. 2025Training and Inference Efficiency of Encoder-Decoder Speech Modelscited by 8
  3. 2025Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languagescited by 8
  4. 2024SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generationcited by 6
  5. 2025Word Level Timestamp Generation for Automatic Speech Recognition and Translationcited by 6
  6. 2025Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnetcited by 6
  7. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Expertscited by 5
  8. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 5
  9. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  10. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  11. 2025OLMoASR: Open Models and Data for Training Robust Speech Recognition Modelscited by 4
  12. 2025SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASRcited by 4
  13. 2024Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Promptingcited by 3
  14. 2025Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired Peoplecited by 3
  15. 2025Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech datacited by 3
  16. 2024Clean Label Attacks against SLU Systemscited by 2
  17. 2025FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italiancited by 2
  18. 2025Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challengescited by 2
  19. 2025CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguesecited by 2
  20. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  21. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  22. 2025WST: Weakly Supervised Transducer for Automatic Speech Recognitioncited by 1
  23. 2026SSVD-O: Parameter-Efficient Fine-Tuning with Structured SVD for Speech Recognitioncited by 1
  24. 2026PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmarkcited by 1
  25. 2026WorldSpeech: A Multilingual Speech Corpus from Around the Worldcited by 1
  26. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  27. 2026Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancementcited by 0
  28. 2026Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?cited by 0
  29. 2026Context-Aware Interleaved Batching for WhisperXcited by 0
  30. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0