mlsanityData to 7 Oct 2026

Attention-based Contextual Language Model Adaptation for Speech Recognition

arXiv 2106.01451 ↗First posted 2 Jun 2021cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Language modeling (LM) for automatic speech recognition (ASR) does not usually incorporate utterance level contextual information. For some domains like voice assistants, however, additional context, such as the time at which an utterance was spoken, provides a rich input signal. We introduce an attention mechanism for training neural speech recognition language models on both text and non-linguistic contextual data. When applied to a large de-identified dataset of utterances collected by a popular voice assistant platform, our method reduces perplexity by 7.0% relative over a standard LM that does not incorporate contextual information. When evaluated on utterances extracted from the long tail of the dataset, our method improves perplexity by 9.0% relative over a standard LM and by over 2.8% relative when compared to a state-of-the-art model for contextual LM.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

16 of 29 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  6. 2020Contextual RNN-T For Open Domain ASRcited by 85
  7. 2016Aspect Level Sentiment Classification with Deep Memory Networkcited by 82
  8. 2019Personalized Dialogue Generation with Diversified Traitscited by 73
  9. 2017Generative and Discriminative Text Classification with Recurrent Neural Networkscited by 30
  10. 2018Personalized Language Model for Query Auto-Completioncited by 25
  11. 2017Neural Personalized Response Generation as Domain Adaptationcited by 19
  12. 2016Contextual LSTM (CLSTM) models for Large scale NLP taskscited by 15
4 more
  1. 2017Factorization tricks for LSTM networkscited by 13
  2. 2020Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Modelcited by 11
  3. 2017Low-Rank RNN Adaptation for Context-Aware Language Modelingcited by 8
  4. 2017Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Networkcited by 5

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1.

4 papers.

  1. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  2. 2022Arithmetic Sampling: Parallel Diverse Decoding for Large Language Modelscited by 18
  3. 2022CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signalscited by 5
  4. 2021Improving RNN-T ASR Performance with Date-Time and Location Awarenesscited by 2