# Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition

arXiv: https://arxiv.org/abs/2109.00627 · HTML: https://mlsanity.com/paper/2109.00627/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017)
- [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020)
- [Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversations](https://mlsanity.com/paper/1910.01335/) (2019)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Improved training of end-to-end attention models for speech recognition](https://mlsanity.com/paper/1805.03294/) (2018)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Back-Translation-Style Data Augmentation for End-to-End ASR](https://mlsanity.com/paper/1807.10893/) (2018)
- [Contextual Speech Recognition with Difficult Negative Training Examples](https://mlsanity.com/paper/1810.12170/) (2018)
- [End-to-end contextual speech recognition using class language models and a token passing decoder](https://mlsanity.com/paper/1812.02142/) (2018)
- [Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.02720/) (2017)

## Built on by (most cited first, 30 of 32)

- [Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network](https://mlsanity.com/paper/2305.12493/) (2023)
- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [Towards Contextual Spelling Correction for Customization of End-to-end Speech Recognition Systems](https://mlsanity.com/paper/2203.00888/) (2022)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model](https://mlsanity.com/paper/2209.01250/) (2022)
- [Phoneme-aware Encoding for Prefix-tree-based Contextual ASR](https://mlsanity.com/paper/2312.09582/) (2023)
- [Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition](https://mlsanity.com/paper/2306.00804/) (2023)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation](https://mlsanity.com/paper/2302.11192/) (2023)
- [Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition](https://mlsanity.com/paper/2304.08862/) (2023)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary](https://mlsanity.com/paper/2405.13344/) (2024)
- [End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator](https://mlsanity.com/paper/2210.16554/) (2022)
- [Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2305.18824/) (2023)
- [Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer](https://mlsanity.com/paper/2311.08966/) (2023)
- [Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation](https://mlsanity.com/paper/2305.17846/) (2023)
- [Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model](https://mlsanity.com/paper/2410.18363/) (2024)
- [Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?](https://mlsanity.com/paper/2210.13189/) (2022)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)
- [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024)
- [Zero-shot Context Biasing with Trie-based Decoding using Synthetic Multi-Pronunciation](https://mlsanity.com/paper/2508.17796/) (2025)
- [Deep CLAS: Deep Contextual Listen, Attend and Spell](https://mlsanity.com/paper/2409.17603/) (2024)
- [An Effective Context-Balanced Adaptation Approach for Long-Tailed Speech Recognition](https://mlsanity.com/paper/2409.06468/) (2024)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary Prediction and Activation](https://mlsanity.com/paper/2505.23077/) (2025)
- [Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems](https://mlsanity.com/paper/2506.06252/) (2025)
- [PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding](https://mlsanity.com/paper/2506.11064/) (2025)
- [Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling](https://mlsanity.com/paper/2509.05908/) (2025)
- [A Neural Model for Contextual Biasing Score Learning and Filtering](https://mlsanity.com/paper/2510.23849/) (2025)

## Abstract (arXiv)

Contextual knowledge is important for real-world automatic speech recognition (ASR) applications. In this paper, a novel tree-constrained pointer generator (TCPGen) component is proposed that incorporates such knowledge as a list of biasing words into both attention-based encoder-decoder and transducer end-to-end ASR models in a neural-symbolic way. TCPGen structures the biasing words into an efficient prefix tree to serve as its symbolic input and creates a neural shortcut between the tree and the final ASR output distribution to facilitate recognising biasing words during decoding. Systems were trained and evaluated on the Librispeech corpus where biasing words were extracted at the scales of an utterance, a chapter, or a book to simulate different application scenarios. Experimental results showed that TCPGen consistently improved word error rates (WERs) compared to the baselines, and in particular, achieved significant WER reductions on the biasing words. TCPGen is highly efficient: it can handle 5,000 biasing words and distractors and only add a small overhead to memory use and computation cost.