# OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary

arXiv: https://arxiv.org/abs/2506.09448 · HTML: https://mlsanity.com/paper/2506.09448/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification](https://mlsanity.com/paper/2402.12654/) (2024)
- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/2107.02268/) (2021)
- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Search](https://mlsanity.com/paper/2401.10449/) (2024)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Phoneme-aware Encoding for Prefix-tree-based Contextual ASR](https://mlsanity.com/paper/2312.09582/) (2023)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary](https://mlsanity.com/paper/2405.13344/) (2024)
- [Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer](https://mlsanity.com/paper/2311.08966/) (2023)
- [VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning](https://mlsanity.com/paper/2410.17485/) (2024)
- [Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation](https://mlsanity.com/paper/2305.17846/) (2023)
- [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023)
- [InterBiasing: Boost Unseen Word Recognition through Biasing Intermediate Predictions](https://mlsanity.com/paper/2406.14890/) (2024)
- [Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model](https://mlsanity.com/paper/2410.18363/) (2024)

## Built on by (most cited first, 4 of 4)

- [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://mlsanity.com/paper/2602.12911/) (2026)
- [When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR](https://mlsanity.com/paper/2605.28211/) (2026)
- [How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs](https://mlsanity.com/paper/2608.05759/) (2026)
- [Context Biasing for Pronunciation-Orthography Mismatch in Automatic Speech Recognition](https://mlsanity.com/paper/2506.18703/) (2025)

## Abstract (arXiv)

Speech foundation models (SFMs), such as Open Whisper-Style Speech Models (OWSM), are trained on massive datasets to achieve accurate automatic speech recognition. However, even SFMs struggle to accurately recognize rare and unseen words. While contextual biasing (CB) is a promising approach to improve recognition of such words, most CB methods are trained from scratch, resulting in lower performance than SFMs due to the lack of pre-trained knowledge. This paper integrates an existing CB method with OWSM v3.1 while freezing its pre-trained parameters. By leveraging the knowledge embedded in SFMs, the proposed method enables effective CB while preserving the advantages of SFMs, even with a small dataset. Experimental results show that the proposed method improves the biasing word error rate (B-WER) by 11.6 points, resulting in a 0.9 point improvement in the overall WER while reducing the real-time factor by 7.5% compared to the non-biasing baseline on the LibriSpeech 100 test-clean set.