# Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understanding

arXiv: https://arxiv.org/abs/2112.06743 · HTML: https://mlsanity.com/paper/2112.06743/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 13 Dec 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Dialog-context aware end-to-end speech recognition](https://mlsanity.com/paper/1808.02171/) (2018)
- [Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models](https://mlsanity.com/paper/1910.09463/) (2019)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [Exploring Transfer Learning For End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2012.08549/) (2020)
- [Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion](https://mlsanity.com/paper/1906.11604/) (2019)
- [Cross-Attention End-to-End ASR for Two-Party Conversations](https://mlsanity.com/paper/1907.10726/) (2019)
- [Not All Attention Is Needed: Gated Attention Network for Sequence Data](https://mlsanity.com/paper/1912.00349/) (2019)
- [Contextual Language Model Adaptation for Conversational Agents](https://mlsanity.com/paper/1806.10215/) (2018)
- [CASA-NLU: Context-Aware Self-Attentive Natural Language Understanding for Task-Oriented Chatbots](https://mlsanity.com/paper/1909.08705/) (2019)
- [Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding](https://mlsanity.com/paper/2102.06750/) (2021)
- [Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-End](https://mlsanity.com/paper/2105.07071/) (2021)
- [An Efficient Approach to Encoding Context for Spoken Language Understanding](https://mlsanity.com/paper/1807.00267/) (2018)
- [Transfer Learning for Context-Aware Spoken Language Understanding](https://mlsanity.com/paper/2003.01305/) (2020)
- [Dynamically Context-Sensitive Time-Decay Attention for Dialogue Modeling](https://mlsanity.com/paper/1809.01557/) (2018)

## Built on by (most cited first, 5 of 5)

- [PromptASR for contextualized ASR with controllable style](https://mlsanity.com/paper/2309.07414/) (2023)
- [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023)
- [Promptformer: Prompted Conformer Transducer for ASR](https://mlsanity.com/paper/2401.07360/) (2024)
- [Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2401.02417/) (2024)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)

## Abstract (arXiv)

Recent years have seen significant advances in end-to-end (E2E) spoken language understanding (SLU) systems, which directly predict intents and slots from spoken audio. While dialogue history has been exploited to improve conventional text-based natural language understanding systems, current E2E SLU approaches have not yet incorporated such critical contextual signals in multi-turn and task-oriented dialogues. In this work, we propose a contextual E2E SLU model architecture that uses a multi-head attention mechanism over encoded previous utterances and dialogue acts (actions taken by the voice assistant) of a multi-turn dialogue. We detail alternative methods to integrate these contexts into the state-ofthe-art recurrent and transformer-based models. When applied to a large de-identified dataset of utterances collected by a voice assistant, our method reduces average word and semantic error rates by 10.8% and 12.6%, respectively. We also present results on a publicly available dataset and show that our method significantly improves performance over a noncontextual baseline