# Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model

arXiv: https://arxiv.org/abs/2005.07394 · HTML: https://mlsanity.com/paper/2005.07394/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2020.

## Built on by (most cited first, 11 of 11)

- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling](https://mlsanity.com/paper/2106.09532/) (2021)
- [Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2305.18824/) (2023)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Attention-based Contextual Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2106.01451/) (2021)
- [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)
- [Efficient Trie-based Biasing using K-step Prediction for Rare Word Recognition](https://mlsanity.com/paper/2509.09196/) (2025)

## Abstract (arXiv)

Videos uploaded on social media are often accompanied with textual descriptions. In building automatic speech recognition (ASR) systems for videos, we can exploit the contextual information provided by such video metadata. In this paper, we explore ASR lattice rescoring by selectively attending to the video descriptions. We first use an attention based method to extract contextual vector representations of video metadata, and use these representations as part of the inputs to a neural language model during lattice rescoring. Secondly, we propose a hybrid pointer network approach to explicitly interpolate the word probabilities of the word occurrences in metadata. We perform experimental evaluations on both language modeling and ASR tasks, and demonstrate that both proposed methods provide performance improvements by selectively leveraging the video metadata.