mlsanityData to 7 Oct 2026

Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition

arXiv 2306.16007 ↗First posted 28 Jun 2023cs.CL eess.AS eess.SPcited by 48

Not on the map. Why?

Abstract (arXiv)

The integration of Language Models (LMs) has proven to be an effective way to address domain shifts in speech recognition. However, these approaches usually require a significant amount of target domain text data for the training of LMs. Different from these methods, in this work, with only a domain-specific text prompt, we propose two zero-shot ASR domain adaptation methods using LLaMA, a 7-billion-parameter large language model (LLM). LLM is used in two ways: 1) second-pass rescoring: reranking N-best hypotheses of a given ASR system with LLaMA; 2) deep LLM-fusion: incorporating LLM into the decoder of an encoder-decoder based ASR system. Experiments show that, with only one domain prompt, both methods can effectively reduce word error rates (WER) on out-of-domain TedLium-2 and SPGISpeech datasets. Especially, the deep LLM-fusion has the advantage of better recall of entity and out-of-vocabulary words.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

30 of 40 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
18 more
  1. 2015Attention-Based Models for Speech Recognitioncited by 400
  2. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  3. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  4. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  5. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  6. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  7. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  8. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 77
  9. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  10. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  11. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 31
  12. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  13. 2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
  14. 2021Adapting Long Context NLM for ASR Rescoring in Conversational Agentscited by 12
  15. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
  16. 2023Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmaxcited by 8
  17. 2021ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modelingcited by 7
  18. 2022Residual Language Model for End-to-end Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 19; 2025: 12; 2026: 10.

48 papers.

Sort:

By area:

  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  3. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  4. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  5. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  6. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  7. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  8. 2024Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Modelscited by 29
  9. 2023PromptASR for contextualized ASR with controllable stylecited by 18
  10. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  11. 2024When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detectioncited by 16
  12. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Modelscited by 15
36 more
  1. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  2. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  3. 2024Contextualization of ASR with LLM using phonetic retrieval-based augmentationcited by 10
  4. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  5. 2026Text-only adaptation in LLM-based ASR through text denoisingcited by 8
  6. 2023Contextual Biasing of Named-Entities with Large Language Modelscited by 7
  7. 2025Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuningcited by 7
  8. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  9. 2024Improving Domain-Specific ASR with LLM-Generated Contextual Descriptionscited by 6
  10. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Modelscited by 5
  11. 2024LLMs for Test Input Generation for Semantic Cachescited by 4
  12. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  13. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  14. 2024Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Trainingcited by 3
  15. 2024Large Language Model Should Understand Pinyin for Chinese ASR Error Correctioncited by 3
  16. 2024LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliterationcited by 3
  17. 2025Towards Reliable Large Audio Language Modelcited by 3
  18. 2026IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languagescited by 3
  19. 2023Improved Factorized Neural Transducer Model For text-only Domain Adaptationcited by 2
  20. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challengescited by 2
  21. 2025End-to-end Automatic Speech Recognition and Speech Translation: Integration of Speech Foundational Models and LLMscited by 2
  22. 2026Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projectioncited by 2
  23. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsamplingcited by 2
  24. 2024Ayn: A Tiny yet Competitive Indian Legal Language Model Pretrained from Scratchcited by 1
  25. 2024A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASRcited by 1
  26. 2025Customizing Speech Recognition Model with Large Language Model Feedbackcited by 1
  27. 2025Improving Contextual ASR via Multi-grained Fusion with Large Language Modelscited by 1
  28. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  29. 2025WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformerscited by 0
  30. 2025MEC3O: Multi-Expert Consensus for Code Time Complexity Predictioncited by 0
  31. 2026Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformerscited by 0
  32. 2026ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generationcited by 0
  33. 2026Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASRcited by 0
  34. 2026Merging the Knowledge of LLMs for Automatic Speech Recognitioncited by 0
  35. 2026HearInContext: A Benchmark for Implicit Context in Speech Recognitioncited by 0
  36. 2026agentic-ger: terminology recovery in long-form speech using global contextcited by 0