# Attention-based Contextual Language Model Adaptation for Speech Recognition

arXiv: https://arxiv.org/abs/2106.01451 · HTML: https://mlsanity.com/paper/2106.01451/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Aspect Level Sentiment Classification with Deep Memory Network](https://mlsanity.com/paper/1605.08900/) (2016)
- [Personalized Dialogue Generation with Diversified Traits](https://mlsanity.com/paper/1901.09672/) (2019)
- [Generative and Discriminative Text Classification with Recurrent Neural Networks](https://mlsanity.com/paper/1703.01898/) (2017)
- [Personalized Language Model for Query Auto-Completion](https://mlsanity.com/paper/1804.09661/) (2018)
- [Neural Personalized Response Generation as Domain Adaptation](https://mlsanity.com/paper/1701.02073/) (2017)
- [Contextual LSTM (CLSTM) models for Large scale NLP tasks](https://mlsanity.com/paper/1602.06291/) (2016)
- [Factorization tricks for LSTM networks](https://mlsanity.com/paper/1703.10722/) (2017)
- [Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model](https://mlsanity.com/paper/2005.07394/) (2020)
- [Low-Rank RNN Adaptation for Context-Aware Language Modeling](https://mlsanity.com/paper/1710.02603/) (2017)
- [Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network](https://mlsanity.com/paper/1701.03578/) (2017)

## Built on by (most cited first, 4 of 4)

- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models](https://mlsanity.com/paper/2210.15458/) (2022)
- [CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals](https://mlsanity.com/paper/2203.08774/) (2022)
- [Improving RNN-T ASR Performance with Date-Time and Location Awareness](https://mlsanity.com/paper/2106.06183/) (2021)

## Abstract (arXiv)

Language modeling (LM) for automatic speech recognition (ASR) does not usually incorporate utterance level contextual information. For some domains like voice assistants, however, additional context, such as the time at which an utterance was spoken, provides a rich input signal. We introduce an attention mechanism for training neural speech recognition language models on both text and non-linguistic contextual data. When applied to a large de-identified dataset of utterances collected by a popular voice assistant platform, our method reduces perplexity by 7.0% relative over a standard LM that does not incorporate contextual information. When evaluated on utterances extracted from the long tail of the dataset, our method improves perplexity by 9.0% relative over a standard LM and by over 2.8% relative when compared to a state-of-the-art model for contextual LM.