# Leveraging Cross-Utterance Context For ASR Decoding

arXiv: https://arxiv.org/abs/2306.16903 · HTML: https://mlsanity.com/paper/2306.16903/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention](https://mlsanity.com/paper/2108.00154/) (2021)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Shortformer: Better Language Modeling using Shorter Inputs](https://mlsanity.com/paper/2012.15832/) (2020)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization](https://mlsanity.com/paper/2101.11075/) (2021)
- [Transformer Language Models with LSTM-based Cross-utterance Information Representation](https://mlsanity.com/paper/2102.06474/) (2021)
- [Online Continual Learning of End-to-End Speech Recognition Models](https://mlsanity.com/paper/2207.05071/) (2022)
- [Comparison of Decoding Strategies for CTC Acoustic Models](https://mlsanity.com/paper/1708.04469/) (2017)
- [An Empirical Study of Efficient ASR Rescoring with Transformers](https://mlsanity.com/paper/1910.11450/) (2019)
- [Long-span language modeling for speech recognition](https://mlsanity.com/paper/1911.04571/) (2019)
- [Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition](https://mlsanity.com/paper/2106.06922/) (2021)
- [Character-Level Incremental Speech Recognition with Recurrent Neural Networks](https://mlsanity.com/paper/1601.06581/) (2016)
- [Lattention: Lattice-attention in ASR rescoring](https://mlsanity.com/paper/2111.10157/) (2021)
- [LSTM-LM with Long-Term History for First-Pass Decoding in Conversational Speech Recognition](https://mlsanity.com/paper/2010.11349/) (2020)

## Built on by (most cited first, 1 of 1)

- [How Much Context Does My Attention-Based ASR System Need?](https://mlsanity.com/paper/2310.15672/) (2023)

## Abstract (arXiv)

While external language models (LMs) are often incorporated into the decoding stage of automated speech recognition systems, these models usually operate with limited context. Cross utterance information has been shown to be beneficial during second pass re-scoring, however this limits the hypothesis space based on the local information available to the first pass LM. In this work, we investigate the incorporation of long-context transformer LMs for cross-utterance decoding of acoustic models via beam search, and compare against results from n-best rescoring. Results demonstrate that beam search allows for an improved use of cross-utterance context. When evaluating on the long-format dataset AMI, results show a 0.7% and 0.3% absolute reduction on dev and test sets compared to the single-utterance setting, with improvements when including up to 500 tokens of prior context. Evaluations are also provided for Tedlium-1 with less significant improvements of around 0.1% absolute.