mlsanityData to 7 Oct 2026

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization

arXiv 2305.11095 ↗First posted 18 May 2023eess.AS cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering. We selected three tasks: audio-visual speech recognition (AVSR), code-switched speech recognition (CS-ASR), and speech translation (ST) on unseen language pairs. We design task-specific prompts, by either leveraging another large-scale model, or simply manipulating the special tokens in the default prompts. Experiments show that compared to the default prompts, our proposed prompts improve performance by 10% to 45% on the three zero-shot tasks, and even outperform SotA supervised models on some datasets. In addition, our experiments reveal many interesting properties of Whisper, including its robustness to prompts, bias on accents, and the multilingual understanding in its latent space. Code is available at https://github.com/jasonppy/PromptingWhisper

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

31 of 37 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
19 more
  1. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  4. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  5. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  6. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  7. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 43
  8. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  9. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  10. 2019Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learningcited by 25
  11. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 25
  12. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  13. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 12
  14. 2022Simple and Effective Unsupervised Speech Translationcited by 12
  15. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  16. 2020Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoderscited by 10
  17. 2018Towards Unsupervised Speech-to-Text Translationcited by 6
  18. 2022Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generationcited by 5
  19. 2022Integrated Parameter-Efficient Tuning for General-Purpose Audio Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 16; 2025: 11; 2026: 4.

43 papers.

Sort:

By area:

  1. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  2. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  3. 2023Can Whisper perform speech-based in-context learning?cited by 35
  4. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  5. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  6. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  7. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  8. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  9. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  10. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  11. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  12. 2024Cross-Lingual Transfer Learning for Speech Translationcited by 11
31 more
  1. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  2. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10
  3. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  4. 2023Instruction-Following Speech Recognitioncited by 7
  5. 2023Generative error correction for code-switching speech recognition using large language modelscited by 7
  6. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 7
  7. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  8. 2023Adapting the adapters for code-switching in multilingual ASRcited by 6
  9. 2024SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognitioncited by 6
  10. 2024Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systemscited by 5
  11. 2024Keyword-Guided Adaptation of Automatic Speech Recognitioncited by 5
  12. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Expertscited by 5
  13. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  14. 2024Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modelingcited by 4
  15. 2025Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slidescited by 4
  16. 2025Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Reviewcited by 4
  17. 2025Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talkscited by 4
  18. 2024SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Datacited by 3
  19. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semanticallycited by 3
  20. 2024Finetuning End-to-End Models for Estonian Conversational Spoken Language Translationcited by 1
  21. 2024Romanization Encoding For Multilingual ASRcited by 1
  22. 2025Probing the Hidden Talent of ASR Foundation Models for L2 English Oral Assessmentcited by 1
  23. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  24. 2025Optimizing ASR for Catalan-Spanish Code-Switching: A Comparative Analysis of Methodologiescited by 0
  25. 2025What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systemscited by 0
  26. 2025Refining Transcripts With TV Subtitles by Prompt-Based Weakly Supervised Training of ASRcited by 0
  27. 2025Whisper-UT: A Unified Translation Framework for Speech and Textcited by 0
  28. 2026Towards Comprehensive Semantic Speech Embeddings for Chinese Dialectscited by 0
  29. 2026OCR-Enhanced Multimodal ASR Can Read While Listeningcited by 0
  30. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  31. 2026Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languagescited by 0