# Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition

arXiv: https://arxiv.org/abs/2207.00857 · HTML: https://mlsanity.com/paper/2207.00857/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 2 Jul 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020)
- [Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection](https://mlsanity.com/paper/2201.12806/) (2022)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/2107.02268/) (2021)
- [Contextual Speech Recognition with Difficult Negative Training Examples](https://mlsanity.com/paper/1810.12170/) (2018)
- [End-to-end contextual speech recognition using class language models and a token passing decoder](https://mlsanity.com/paper/1812.02142/) (2018)
- [Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model](https://mlsanity.com/paper/2005.07394/) (2020)
- [Contextual Language Model Adaptation for Conversational Agents](https://mlsanity.com/paper/1806.10215/) (2018)
- [Hierarchical Character Embeddings: Learning Phonological and Semantic Representations in Languages of Logographic Origin using Recursive Neural Networks](https://mlsanity.com/paper/1912.09913/) (2019)
- [Fast and Robust Unsupervised Contextual Biasing for Speech Recognition](https://mlsanity.com/paper/2005.01677/) (2020)
- [Neural-FST Class Language Model for End-to-End Speech Recognition](https://mlsanity.com/paper/2201.11867/) (2022)
- [Recursive Neural Network Based Preordering for English-to-Japanese Machine Translation](https://mlsanity.com/paper/1805.10187/) (2018)

## Built on by (most cited first, 17 of 17)

- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [CTC-Assisted LLM-Based Contextual ASR](https://mlsanity.com/paper/2411.06437/) (2024)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correction](https://mlsanity.com/paper/2310.05129/) (2023)
- [End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator](https://mlsanity.com/paper/2210.16554/) (2022)
- [Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2305.18824/) (2023)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks](https://mlsanity.com/paper/2510.13979/) (2025)
- [M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset](https://mlsanity.com/paper/2403.14168/) (2024)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)
- [Zero-shot Context Biasing with Trie-based Decoding using Synthetic Multi-Pronunciation](https://mlsanity.com/paper/2508.17796/) (2025)
- [wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech](https://mlsanity.com/paper/2408.04174/) (2024)
- [PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding](https://mlsanity.com/paper/2506.11064/) (2025)
- [CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models](https://mlsanity.com/paper/2506.12059/) (2025)
- [Improving Contextual ASR via Multi-grained Fusion with Large Language Models](https://mlsanity.com/paper/2507.12252/) (2025)
- [OCR-Enhanced Multimodal ASR Can Read While Listening](https://mlsanity.com/paper/2601.18393/) (2026)
- [PARCO: Phoneme-Augmented Robust Contextual ASR via Contrastive Entity Disambiguation](https://mlsanity.com/paper/2509.04357/) (2025)

## Abstract (arXiv)

Incorporating biasing words obtained as contextual knowledge is critical for many automatic speech recognition (ASR) applications. This paper proposes the use of graph neural network (GNN) encodings in a tree-constrained pointer generator (TCPGen) component for end-to-end contextual ASR. By encoding the biasing words in the prefix-tree with a tree-based GNN, lookahead for future wordpieces in end-to-end ASR decoding is achieved at each tree node by incorporating information about all wordpieces on the tree branches rooted from it, which allows a more accurate prediction of the generation probability of the biasing words. Systems were evaluated on the Librispeech corpus using simulated biasing tasks, and on the AMI corpus by proposing a novel visual-grounded contextual ASR pipeline that extracts biasing words from slides alongside each meeting. Results showed that TCPGen with GNN encodings achieved about a further 15% relative WER reduction on the biasing words compared to the original TCPGen, with a negligible increase in the computation cost for decoding.