mlsanityData to 7 Oct 2026

Can Whisper perform speech-based in-context learning?

arXiv 2309.07081 ↗First posted 13 Sep 2023eess.AS cs.CL cs.SDcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper investigates the in-context learning abilities of the Whisper automatic speech recognition (ASR) models released by OpenAI. A novel speech-based in-context learning (SICL) approach is proposed for test-time adaptation, which can reduce the word error rates (WERs) with only a small number of labelled speech samples without gradient descent. Language-level adaptation experiments using Chinese dialects showed that when applying SICL to isolated word ASR, consistent and considerable relative WER reductions can be achieved using Whisper models of any size on two dialects, which is on average 32.3%. A k-nearest-neighbours-based in-context example selection technique can be applied to further improve the efficiency of SICL, which can increase the average relative WER reduction to 36.4%. The findings are verified using speaker adaptation or continuous speech recognition tasks, and both achieved considerable relative WER reductions. Detailed quantitative analyses are also provided to shed light on SICL's adaptability to phonological variances and dialect-specific lexical nuances.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

29 of 37 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
17 more
  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  2. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  3. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  4. 2021Quantifying Bias in Automatic Speech Recognitioncited by 50
  5. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  6. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  7. 2021Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASRcited by 40
  8. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  9. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  10. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 30
  11. 2021Innovative Bert-based Reranking Language Models for Speech Recognitioncited by 23
  12. 2023Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understandingcited by 21
  13. 2023N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognitioncited by 20
  14. 2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
  15. 2018Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognitioncited by 14
  16. 2023Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Studycited by 7
  17. 2023Towards dialect-inclusive recognition in a low-resource language: are balanced corpora the answer?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 13; 2026: 5.

35 papers.

Sort:

By area:

  1. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  2. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  3. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  4. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  5. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  6. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  7. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  8. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  9. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  10. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  11. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  12. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10
23 more
  1. 2025Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoningcited by 10
  2. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  3. 2024Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalitiescited by 7
  4. 2025In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varietiescited by 7
  5. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  6. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
  7. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  8. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  9. 2024SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognitioncited by 4
  10. 2025TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognitioncited by 4
  11. 2025TICL: Text-Embedding KNN For Speech In-Context Learning Unlocks Speech Recognition Abilities of Large Multimodal Modelscited by 3
  12. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  13. 2026SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectorscited by 2
  14. 2024Using LLM for Real-Time Transcription and Summarization of Doctor-Patient Interactions into ePuskesmas in Indonesia: A Proof-of-Concept Studycited by 1
  15. 2025FlanEC: Exploring Flan-T5 for Post-ASR Error Correctioncited by 1
  16. 2025A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpuscited by 1
  17. 2026MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learningcited by 1
  18. 2026In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Headscited by 1
  19. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  20. 2025Zero-Shot Context-Aware ASR for Diverse Arabic Varietiescited by 0
  21. 2025LLMs for Low-Resource Dialect Translation Using Context-Aware Prompting: A Case Study on Sylheticited by 0
  22. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  23. 2026In-Context Adaptation of Encoder-Decoder Models in Speech Recognitioncited by 0