mlsanityData to 7 Oct 2026

OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer

arXiv 2401.16658 ↗First posted 30 Jan 2024cs.CL eess.AScited by 74

Not on the map. Why?

Abstract (arXiv)

Recent studies have highlighted the importance of fully open foundation models. The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper using public data and open-source toolkits. However, previous versions of OWSM (v1 to v3) are still based on standard Transformer, which might lead to inferior performance compared to state-of-the-art speech encoder architectures. This work aims to improve the performance and efficiency of OWSM without additional data. We present a series of E-Branchformer-based models named OWSM v3.1, ranging from 100M to 1B parameters. OWSM v3.1 outperforms its predecessor, OWSM v3, in most evaluation benchmarks, while showing an improved inference speed of up to 25%. We further reveal the emergent ability of OWSM v3.1 in zero-shot contextual biasing speech recognition. We also provide a model trained on a subset of data with low license restrictions. We will publicly release the code, pre-trained models, and training logs.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
29 more
  1. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  4. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  5. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  9. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  10. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  12. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  13. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  14. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  15. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  16. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  17. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 77
  18. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  19. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  20. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  21. 2022Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Datasetcited by 36
  22. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  23. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  24. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  25. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  26. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  27. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 13
  28. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  29. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 34; 2026: 17.

74 papers.

Sort:

By area:

  1. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  2. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  3. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  4. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  5. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  6. 2024SpoofCeleb: Speech Deepfake Detection and SASV In The Wildcited by 22
  7. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  8. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  9. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  10. 2025CS-FLEURS: A Massively Multilingual and Code-Switched Speech Datasetcited by 11
  11. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
  12. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
38 more
  1. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  2. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  3. 2024Wav2Gloss: Generating Interlinear Glossed Text from Speechcited by 8
  4. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  5. 2025Training and Inference Efficiency of Encoder-Decoder Speech Modelscited by 8
  6. 2025Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languagescited by 8
  7. 2024Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Modelscited by 7
  8. 2025ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systemscited by 7
  9. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  10. 2024Neural Blind Source Separation and Diarization for Distant Speech Recognitioncited by 6
  11. 2024Text-To-Speech Synthesis In The Wildcited by 6
  12. 2025Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnetcited by 6
  13. 2024Exploring the limits of decoder-only models trained on public speech recognition corporacited by 5
  14. 2024Lyrics Transcription for Humans: A Readability-Aware Benchmarkcited by 5
  15. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Expertscited by 5
  16. 2025Selective Attention Merging for low resource tasks: A case study of Child ASRcited by 5
  17. 2025AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Taskscited by 5
  18. 2025PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancementcited by 5
  19. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  20. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  21. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  22. 2025OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabularycited by 4
  23. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 4
  24. 2025OLMoASR: Open Models and Data for Training Robust Speech Recognition Modelscited by 4
  25. 2025The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varietiescited by 4
  26. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  27. 2024Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Promptingcited by 3
  28. 2025Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired Peoplecited by 3
  29. 2025Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech datacited by 3
  30. 2025SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASRcited by 3
  31. 2025Drax: Speech Recognition with Discrete Flow Matchingcited by 3
  32. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
  33. 2025FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italiancited by 2
  34. 2024Efficiently Identifying Low-Quality Language Subsets in Multilingual Datasets: A Case Study on a Large-Scale Multilingual Audio Datasetcited by 1
  35. 2024Enhancing Audiovisual Speech Recognition through Bifocal Preference Optimizationcited by 1
  36. 2025The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergencecited by 1
  37. 2025Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTCcited by 1
  38. 2025DYNAC: Dynamic Vocabulary based Non-Autoregressive Contextualization for Speech Recognitioncited by 1

50 most cited of 74 All 74 (JSON)