# Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism

arXiv: https://arxiv.org/abs/2207.00883 · HTML: https://mlsanity.com/paper/2207.00883/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 2 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [RealFormer: Transformer Likes Residual Attention](https://mlsanity.com/paper/2012.11747/) (2020)
- [Dialog-context aware end-to-end speech recognition](https://mlsanity.com/paper/1808.02171/) (2018)
- [Simplified Self-Attention for Transformer-based End-to-End Speech Recognition](https://mlsanity.com/paper/2005.10463/) (2020)
- [Cross-Attention End-to-End ASR for Two-Party Conversations](https://mlsanity.com/paper/1907.10726/) (2019)
- [Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation](https://mlsanity.com/paper/2102.07935/) (2021)
- [Document Context Language Models](https://mlsanity.com/paper/1511.03962/) (2015)
- [Building competitive direct acoustics-to-word models for English conversational speech recognition](https://mlsanity.com/paper/1712.03133/) (2017)
- [Speaker-aware speech-transformer](https://mlsanity.com/paper/2001.01557/) (2020)
- [Dialog Context Language Modeling with Recurrent Neural Networks](https://mlsanity.com/paper/1701.04056/) (2017)

## Built on by (most cited first, 4 of 4)

- [LongFNT: Long-form Speech Recognition with Factorized Neural Transducer](https://mlsanity.com/paper/2211.09412/) (2022)
- [Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation](https://mlsanity.com/paper/2310.14278/) (2023)
- [Promptformer: Prompted Conformer Transducer for ASR](https://mlsanity.com/paper/2401.07360/) (2024)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)

## Abstract (arXiv)

Transformer-based models have demonstrated their effectiveness in automatic speech recognition (ASR) tasks and even shown superior performance over the conventional hybrid framework. The main idea of Transformers is to capture the long-range global context within an utterance by self-attention layers. However, for scenarios like conversational speech, such utterance-level modeling will neglect contextual dependencies that span across utterances. In this paper, we propose to explicitly model the inter-sentential information in a Transformer based end-to-end architecture for conversational speech recognition. Specifically, for the encoder network, we capture the contexts of previous speech and incorporate such historic information into current input by a context-aware residual attention mechanism. For the decoder, the prediction of current utterance is also conditioned on the historic linguistic information through a conditional decoder framework. We show the effectiveness of our proposed method on several open-source dialogue corpora and the proposed method consistently improved the performance from the utterance-level Transformer-based ASR models.