# End-to-end contextual speech recognition using class language models and a token passing decoder

arXiv: https://arxiv.org/abs/1812.02142 · HTML: https://mlsanity.com/paper/1812.02142/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2018.

## Built on by (most cited first, 26 of 26)

- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition](https://mlsanity.com/paper/2402.05457/) (2024)
- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [CTC-Assisted LLM-Based Contextual ASR](https://mlsanity.com/paper/2411.06437/) (2024)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model](https://mlsanity.com/paper/2209.01250/) (2022)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [End-to-end contextual asr based on posterior distribution adaptation for hybrid ctc/attention system](https://mlsanity.com/paper/2202.09003/) (2022)
- [End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator](https://mlsanity.com/paper/2210.16554/) (2022)
- [Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2305.18824/) (2023)
- [Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model](https://mlsanity.com/paper/2410.18363/) (2024)
- [Implementing contextual biasing in GPU decoder for online ASR](https://mlsanity.com/paper/2306.15685/) (2023)
- [Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?](https://mlsanity.com/paper/2210.13189/) (2022)
- [A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers](https://mlsanity.com/paper/2304.07842/) (2023)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)
- [Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation](https://mlsanity.com/paper/2306.08588/) (2023)
- [Lessons Learned in ATCO2: 5000 hours of Air Traffic Control Communications for Robust Automatic Speech Recognition and Understanding](https://mlsanity.com/paper/2305.01155/) (2023)
- [Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper](https://mlsanity.com/paper/2409.13499/) (2024)
- [Spelling Correction through Rewriting of Non-Autoregressive ASR Lattices](https://mlsanity.com/paper/2409.16469/) (2024)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary Prediction and Activation](https://mlsanity.com/paper/2505.23077/) (2025)
- [Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling](https://mlsanity.com/paper/2509.05908/) (2025)
- [Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer](https://mlsanity.com/paper/2309.07648/) (2023)

## Abstract (arXiv)

End-to-end modeling (E2E) of automatic speech recognition (ASR) blends all the components of a traditional speech recognition system into a unified model. Although it simplifies training and decoding pipelines, the unified model is hard to adapt when mismatch exists between training and test data. In this work, we focus on contextual speech recognition, which is particularly challenging for E2E models because it introduces significant mismatch between training and test data. To improve the performance in the presence of complex contextual information, we propose to use class-based language models(CLM) that can populate the classes with contextdependent information in real-time. To enable this approach to scale to a large number of class members and minimize search errors, we propose a token passing decoder with efficient token recombination for E2E systems for the first time. We evaluate the proposed system on general and contextual ASR, and achieve relative 62% Word Error Rate(WER) reduction for contextual ASR without hurting performance for general ASR. We show that the proposed method performs well without modification of the decoding hyper-parameters across tasks, making it a general solution for E2E ASR.