mlsanityData to 14 Oct 2026

Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning

arXiv 2307.10274 ↗First posted 18 Jul 2023eess.AS cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a method to create domain-sensitive speech recognition models that utilize textual domain information by conditioning its generation on a given text prompt. This is accomplished by fine-tuning a pre-trained, end-to-end model (Whisper) to learn from demonstrations with prompt examples. We show that this ability can be generalized to different domains and even various prompt contexts, with our model gaining a Word Error Rate (WER) reduction of up to 33% on unseen datasets from various domains, such as medical conversation, air traffic control communication, and financial meetings. Considering the limited availability of audio-transcript pair data, we further extend our method to text-only fine-tuning to achieve domain sensitivity as well as domain adaptation. We demonstrate that our text-only fine-tuned model can also attend to various prompt contexts, with the model reaching the most WER reduction of 29% on the medical conversation dataset.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

28 of 39 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
16 more
  1. 2015Attention-Based Models for Speech Recognitioncited by 401
  2. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  3. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  4. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  5. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  6. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 50
  7. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  8. 2021Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognitioncited by 25
  9. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  10. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  11. 2021Domain-aware Neural Language Models for Speech Recognitioncited by 13
  12. 2021Improving callsign recognition with air-surveillance data in air-traffic communicationcited by 11
  13. 2023CB-Conformer: Contextual biasing Conformer for biased word recognitioncited by 11
  14. 2023Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmaxcited by 8
  15. 2022Residual Language Model for End-to-end Speech Recognitioncited by 5
  16. 2021Detecting English Speech in the Air Traffic Control Voice Communicationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 18
  2. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  3. 2025Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuningcited by 7
  4. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  5. 2024Keyword-Guided Adaptation of Automatic Speech Recognitioncited by 5
  6. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  7. 2024Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modelingcited by 4
  8. 2024WhisperNER: Unified Open Named Entity and Speech Recognitioncited by 3
  9. 2025Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotationcited by 1
  10. 2025Refining Transcripts With TV Subtitles by Prompt-Based Weakly Supervised Training of ASRcited by 0
  11. 2026TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMscited by 0
  12. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
1 more
  1. 2026Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactionscited by 0