mlsanityData to 14 Oct 2026

End-to-End Speech Recognition Contextualization with Large Language Models

arXiv 2309.10917 ↗First posted 19 Sep 2023eess.AS cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

In recent years, Large Language Models (LLMs) have garnered significant attention from the research community due to their exceptional performance and generalization capabilities. In this paper, we introduce a novel method for contextualizing speech recognition models incorporating LLMs. Our approach casts speech recognition as a mixed-modal language modeling task based on a pretrained LLM. We provide audio features, along with optional text tokens for context, to train the system to complete transcriptions in a decoder-only fashion. As a result, the system is implicitly incentivized to learn how to leverage unstructured contextual information during training. Our empirical results demonstrate a significant improvement in performance, with a 6% WER reduction when additional textual context is provided. Moreover, we find that our method performs competitively and improve by 7.5% WER overall and 17% WER on rare words against a baseline contextualized RNN-T system that has been trained on more than twenty five times larger speech dataset. Overall, we demonstrate that by only adding a handful number of trainable parameters via adapters, we can unlock contextualized speech recognition capability for the pretrained LLM while keeping the same text-only input functionality.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

22 of 27 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  9. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  10. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  11. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  12. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
10 more
  1. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  2. 2018Deep context: end-to-end contextual speech recognitioncited by 137
  3. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
  4. 2020Contextual RNN-T For Open Domain ASRcited by 85
  5. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  6. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  7. 2023Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognitioncited by 18
  8. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 16
  9. 2023Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognitioncited by 12
  10. 2019G2G: TTS-Driven Pronunciation Learning for Graphemic Hybrid ASRcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 8; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024Large Language Models for Time Series: A Surveycited by 114
  2. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  3. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  4. 2024Efficient Streaming LLM for Speech Recognitioncited by 12
  5. 2024Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbationcited by 11
  6. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  7. 2024Contextualization of ASR with LLM using phonetic retrieval-based augmentationcited by 10
  8. 2025Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuningcited by 7
  9. 2024LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generationcited by 4
  10. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  11. 2025WCTC-Biasing: Retraining-free Contextual Biasing ASR with Wildcard CTC-based Keyword Spotting and Inter-layer Biasingcited by 3
  12. 2025SepALM: Audio Language Models Are Error Correctors for Robust Speech Separationcited by 2
10 more
  1. 2024Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictionscited by 1
  2. 2024Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Wordscited by 1
  3. 2025Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishingcited by 1
  4. 2025Lightweight Prompt Biasing for Contextualized End-to-End ASR Systemscited by 1
  5. 2025PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognitioncited by 1
  6. 2025A Neural Model for Contextual Biasing Score Learning and Filteringcited by 1
  7. 2026How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMscited by 1
  8. 2025Exploring the Feasibility of End-to-End Large Language Model as a Compilercited by 0
  9. 2026Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Modelscited by 0
  10. 2026Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?cited by 0