# CTC-Assisted LLM-Based Contextual ASR

arXiv: https://arxiv.org/abs/2411.06437 · HTML: https://mlsanity.com/paper/2411.06437/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Builds on (most cited first, 17 of 17)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020)
- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [End-to-end contextual speech recognition using class language models and a token passing decoder](https://mlsanity.com/paper/1812.02142/) (2018)
- [CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2012.09466/) (2020)
- [Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Search](https://mlsanity.com/paper/2401.10449/) (2024)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Improving ASR Contextual Biasing with Guided Attention](https://mlsanity.com/paper/2401.08835/) (2024)
- [Phoneme-aware Encoding for Prefix-tree-based Contextual ASR](https://mlsanity.com/paper/2312.09582/) (2023)
- [Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducer](https://mlsanity.com/paper/2301.06735/) (2023)

## Built on by (most cited first, 19 of 19)

- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition](https://mlsanity.com/paper/2505.22251/) (2025)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [Zero-shot Context Biasing with Trie-based Decoding using Synthetic Multi-Pronunciation](https://mlsanity.com/paper/2508.17796/) (2025)
- [Boosting CTC-Based ASR Using LLM-Based Intermediate Loss Regularization](https://mlsanity.com/paper/2506.22846/) (2025)
- [Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems](https://mlsanity.com/paper/2506.06252/) (2025)
- [CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models](https://mlsanity.com/paper/2506.12059/) (2025)
- [Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling](https://mlsanity.com/paper/2509.05908/) (2025)
- [PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognition](https://mlsanity.com/paper/2509.12647/) (2025)
- [When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR](https://mlsanity.com/paper/2605.28211/) (2026)
- [MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond](https://mlsanity.com/paper/2607.22100/) (2026)
- [Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integration](https://mlsanity.com/paper/2502.16142/) (2025)
- [H-PRM: A Pluggable Hotword Pre-Retrieval Module for Various Speech Recognition Systems](https://mlsanity.com/paper/2508.18295/) (2025)
- [RASST: Retrieval-Augmented Simultaneous Speech Translation](https://mlsanity.com/paper/2601.22777/) (2026)
- [CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR](https://mlsanity.com/paper/2601.22792/) (2026)
- [Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning](https://mlsanity.com/paper/2603.06505/) (2026)
- [COALA: Robust Contextualized Speech-augmented Language Modeling for ASR via Contrastive Regularizer and Biasing Score Estimation](https://mlsanity.com/paper/2607.08117/) (2026)
- [PTC-Bias: Phoneme-Level Temporal Competition for Bias Retrieval and Post-Decoding Correction in Speech LLMs](https://mlsanity.com/paper/2609.28727/) (2026)
- [LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR](https://mlsanity.com/paper/2610.16579/) (2026)

## Abstract (arXiv)

Contextual ASR or hotword customization holds substantial practical value. Despite the impressive performance of current end-to-end (E2E) automatic speech recognition (ASR) systems, they often face challenges in accurately recognizing rare words. Typical E2E contextual ASR models commonly feature complex architectures and decoding mechanisms, limited in performance and susceptible to interference from distractor words. With large language model (LLM)-based ASR models emerging as the new mainstream, we propose a CTC-Assisted LLM-Based Contextual ASR model with an efficient filtering algorithm. By using coarse CTC decoding results to filter potential relevant hotwords and incorporating them into LLM prompt input, our model attains WER/B-WER of 1.27%/3.67% and 2.72%/8.02% on the Librispeech test-clean and test-other sets targeting on recognizing rare long-tail words, demonstrating significant improvements compared to the baseline LLM-based ASR model, and substantially surpassing other related work. More remarkably, with the help of the large language model and proposed filtering algorithm, our contextual ASR model still performs well with 2000 biasing words.