mlsanityData to 7 Oct 2026

Keyword-Guided Adaptation of Automatic Speech Recognition

arXiv 2406.02649 ↗First posted 4 Jun 2024eess.AS cs.LG cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Automatic Speech Recognition (ASR) technology has made significant progress in recent years, providing accurate transcription across various domains. However, some challenges remain, especially in noisy environments and specialized jargon. In this paper, we propose a novel approach for improved jargon word recognition by contextual biasing Whisper-based models. We employ a keyword spotting model that leverages the Whisper encoder representation to dynamically generate prompts for guiding the decoder during the transcription process. We introduce two approaches to effectively steer the decoder towards these prompts: KG-Whisper, which is aimed at fine-tuning the Whisper decoder, and KG-Whisper-PT, which learns a prompt prefix. Our results show a significant improvement in the recognition accuracy of specified keywords and in reducing the overall word error rates. Specifically, in unseen language generalization, we demonstrate an average WER improvement of 5.1% over Whisper.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

26 of 32 references.

References · 26

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  6. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  7. 2018Deep context: end-to-end contextual speech recognitioncited by 136
  8. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  9. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  10. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  11. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  12. 2021Context-Aware Transformer Transducer for Speech Recognitioncited by 56
14 more
  1. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  2. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  3. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  4. 2018Contextual Speech Recognition with Difficult Negative Training Examplescited by 28
  5. 2023Can Contextual Biasing Remain Effective with Whisper and GPT-2?cited by 21
  6. 2022Learning Audio-Text Agreement for Open-vocabulary Keyword Spottingcited by 20
  7. 2023Matching Latent Encoding for Audio-Text based Keyword Spottingcited by 15
  8. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  9. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  10. 2023CB-Conformer: Contextual biasing Conformer for biased word recognitioncited by 11
  11. 2023Flexible Keyword Spotting based on Homogeneous Audio-Text Embeddingcited by 10
  12. 2023Open-vocabulary Keyword-spotting with Adaptive Instance Normalizationcited by 8
  13. 2023A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spottingcited by 5
  14. 2023Combining Language Models For Specialized Domains: A Colorful Approachcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024WhisperNER: Unified Open Named Entity and Speech Recognitioncited by 3
  2. 2025Improving Synthetic Data Training for Contextual Biasing Models with a Keyword-Aware Cost Functioncited by 1
  3. 2025Efficient Trie-based Biasing using K-step Prediction for Rare Word Recognitioncited by 0
  4. 2026Massive Open-Vocabulary Keyword Spottingcited by 0
  5. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0