# Integrating Dialog History into End-to-End Spoken Language Understanding Systems

arXiv: https://arxiv.org/abs/2108.08405 · HTML: https://mlsanity.com/paper/2108.08405/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [Tracking the World State with Recurrent Entity Networks](https://mlsanity.com/paper/1612.03969/) (2016)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study](https://mlsanity.com/paper/1906.01603/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability](https://mlsanity.com/paper/1906.07601/) (2019)
- [End-to-End Spoken Language Understanding Without Full Transcripts](https://mlsanity.com/paper/2009.14386/) (2020)
- [Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation](https://mlsanity.com/paper/1904.08311/) (2019)
- [Improving End-to-End Speech-to-Intent Classification with Reptile](https://mlsanity.com/paper/2008.01994/) (2020)
- [HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking](https://mlsanity.com/paper/1907.00883/) (2019)
- [End-to-end architectures for ASR-free spoken language understanding](https://mlsanity.com/paper/1910.10599/) (2019)
- [Large-scale Transfer Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2008.05671/) (2020)
- [Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems](https://mlsanity.com/paper/2002.06012/) (2020)
- [HarperValleyBank: A Domain-Specific Spoken Dialog Corpus](https://mlsanity.com/paper/2010.13929/) (2020)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [An Efficient Approach to Encoding Context for Spoken Language Understanding](https://mlsanity.com/paper/1807.00267/) (2018)

## Built on by (most cited first, 9 of 9)

- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models](https://mlsanity.com/paper/2202.13155/) (2022)
- [Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding](https://mlsanity.com/paper/2204.05169/) (2022)
- [Two-Pass Low Latency End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.06670/) (2022)
- [Context-aware Fine-tuning of Self-supervised Speech Models](https://mlsanity.com/paper/2212.08542/) (2022)
- [Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History](https://mlsanity.com/paper/2305.00926/) (2023)
- [Is one brick enough to break the wall of spoken dialogue state tracking?](https://mlsanity.com/paper/2311.04923/) (2023)
- [Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Tracking](https://mlsanity.com/paper/2511.22503/) (2025)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)

## Abstract (arXiv)

End-to-end spoken language understanding (SLU) systems that process human-human or human-computer interactions are often context independent and process each turn of a conversation independently. Spoken conversations on the other hand, are very much context dependent, and dialog history contains useful information that can improve the processing of each conversational turn. In this paper, we investigate the importance of dialog history and how it can be effectively integrated into end-to-end SLU systems. While processing a spoken utterance, our proposed RNN transducer (RNN-T) based SLU model has access to its dialog history in the form of decoded transcripts and SLU labels of previous turns. We encode the dialog history as BERT embeddings, and use them as an additional input to the SLU model along with the speech features for the current utterance. We evaluate our approach on a recently released spoken dialog data set, the HarperValleyBank corpus. We observe significant improvements: 8% for dialog action and 30% for caller intent recognition tasks, in comparison to a competitive context independent end-to-end baseline system.