mlsanityData to 14 Oct 2026

Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper

arXiv 2406.05806 ↗First posted 9 Jun 2024cs.CL cs.SD eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

This research explores how the information of prompts interacts with the high-performing speech recognition model, Whisper. We compare its performances when prompted by prompts with correct information and those corrupted with incorrect information. Our results unexpectedly show that Whisper may not understand the textual prompts in a human-expected way. Additionally, we find that performance improvement is not guaranteed even with stronger adherence to the topic information in textual prompts. It is also noted that English prompts generally outperform Mandarin ones on datasets of both languages, likely due to differences in training data distributions for these languages despite the mismatch with pre-training scenarios. Conversely, we discover that Whisper exhibits awareness of misleading information in language tokens by ignoring incorrect language tokens and focusing on the correct ones. In sum, We raise insightful questions about Whisper's prompt understanding and reveal its counter-intuitive behaviors. We encourage further studies.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

36 of 50 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 692
24 more
  1. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  2. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  3. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  4. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  5. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  6. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  7. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  8. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  9. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  10. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 44
  11. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  12. 2023Can Whisper perform speech-based in-context learning?cited by 35
  13. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 26
  14. 2023LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPTcited by 15
  15. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  16. 2024PhoWhisper: Automatic Speech Recognition for Vietnamesecited by 13
  17. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12
  18. 2023Attention-Guided Adaptation for Code-Switching Speech Recognitioncited by 10
  19. 2023VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale F0 Modelingcited by 6
  20. 2023Whisper in Focus: Enhancing Stuttered Speech Classification with Encoder Layer Optimizationcited by 5
  21. 2024Non-autoregressive Sequence-to-Sequence Vision-Language Modelscited by 5
  22. 2024Deconstructing In-Context Learning: Understanding Prompts via Corruptioncited by 4
  23. 2024Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Modelscited by 4
  24. 2024Does ChatGPT and Whisper Make Humanoid Robots More Relatable?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 4.

12 papers.

  1. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  2. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  3. 2025Beyond Transcription: Mechanistic Interpretability in ASRcited by 13
  4. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  5. 2024Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modelingcited by 4
  6. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  7. 2025Adapting Automatic Speech Recognition for Accented Air Traffic Control Communicationscited by 3
  8. 2025Improving Child Speech Recognition and Reading Mistake Detection by Using Promptscited by 3
  9. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  10. 2026IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languagescited by 3
  11. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  12. 2026HearInContext: A Benchmark for Implicit Context in Speech Recognitioncited by 0