# Transformer Language Models with LSTM-based Cross-utterance Information Representation

arXiv: https://arxiv.org/abs/2102.06474 · HTML: https://mlsanity.com/paper/2102.06474/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Adaptive Attention Span in Transformers](https://mlsanity.com/paper/1905.07799/) (2019)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Achieving Human Parity in Conversational Speech Recognition](https://mlsanity.com/paper/1610.05256/) (2016)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [R-Transformer: Recurrent Neural Network Enhanced Transformer](https://mlsanity.com/paper/1907.05572/) (2019)
- [Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1610.09975/) (2016)
- [End-to-end Speech Recognition with Word-based RNN Language Models](https://mlsanity.com/paper/1808.02608/) (2018)
- [Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion](https://mlsanity.com/paper/1906.11604/) (2019)
- [Long-span language modeling for speech recognition](https://mlsanity.com/paper/1911.04571/) (2019)
- [Investigating the Effects of Word Substitution Errors on Sentence Embeddings](https://mlsanity.com/paper/1811.07021/) (2018)
- [Cross-Utterance Language Models with Acoustic Error Sampling](https://mlsanity.com/paper/2009.01008/) (2020)

## Built on by (most cited first, 15 of 15)

- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)
- [ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling](https://mlsanity.com/paper/2106.09532/) (2021)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis](https://mlsanity.com/paper/2412.09032/) (2024)
- [Bayesian Neural Network Language Modeling for Speech Recognition](https://mlsanity.com/paper/2208.13259/) (2022)
- [How Much Context Does My Attention-Based ASR System Need?](https://mlsanity.com/paper/2310.15672/) (2023)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)
- [Leveraging Cross-Utterance Context For ASR Decoding](https://mlsanity.com/paper/2306.16903/) (2023)
- [Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information](https://mlsanity.com/paper/2308.16577/) (2023)
- [Effective Cross-Utterance Language Modeling for Conversational Speech Recognition](https://mlsanity.com/paper/2111.03333/) (2021)
- [A Differential Attention Fusion Model Based on Transformer for Time Series Forecasting](https://mlsanity.com/paper/2202.11402/) (2022)
- [Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems](https://mlsanity.com/paper/2306.14608/) (2023)
- [Lattice Rescoring Based on Large Ensemble of Complementary Neural Language Models](https://mlsanity.com/paper/2312.12764/) (2023)
- [Multi-modal Deepfake Detection and Localization with FPN-Transformer](https://mlsanity.com/paper/2511.08031/) (2025)

## Abstract (arXiv)

The effective incorporation of cross-utterance information has the potential to improve language models (LMs) for automatic speech recognition (ASR). To extract more powerful and robust cross-utterance representations for the Transformer LM (TLM), this paper proposes the R-TLM which uses hidden states in a long short-term memory (LSTM) LM. To encode the cross-utterance information, the R-TLM incorporates an LSTM module together with a segment-wise recurrence in some of the Transformer blocks. In addition to the LSTM module output, a shortcut connection using a fusion layer that bypasses the LSTM module is also investigated. The proposed system was evaluated on the AMI meeting corpus, the Eval2000 and the RT03 telephone conversation evaluation sets. The best R-TLM achieved 0.9%, 0.6%, and 0.8% absolute WER reductions over the single-utterance TLM baseline, and 0.5%, 0.3%, 0.2% absolute WER reductions over a strong cross-utterance TLM baseline on the AMI evaluation set, Eval2000 and RT03 respectively. Improvements on Eval2000 and RT03 were further supported by significance tests. R-TLMs were found to have better LM scores on words where recognition errors are more likely to occur. The R-TLM WER can be further reduced by interpolation with an LSTM-LM.