# Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History

arXiv: https://arxiv.org/abs/2305.00926 · HTML: https://mlsanity.com/paper/2305.00926/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study](https://mlsanity.com/paper/1906.01603/) (2019)
- [End-to-End Monaural Multi-speaker ASR System without Pretraining](https://mlsanity.com/paper/1811.02062/) (2018)
- [Dialogue Act Classification with Context-Aware Self-Attention](https://mlsanity.com/paper/1904.02594/) (2019)
- [Guiding attention in Sequence-to-sequence models for Dialogue Act prediction](https://mlsanity.com/paper/2002.08801/) (2020)
- [Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2011.09044/) (2020)
- [Cross-Attention End-to-End ASR for Two-Party Conversations](https://mlsanity.com/paper/1907.10726/) (2019)
- [End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend](https://mlsanity.com/paper/2102.11525/) (2021)
- [Sequential Dialogue Context Modeling for Spoken Language Understanding](https://mlsanity.com/paper/1705.03455/) (2017)
- [Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems](https://mlsanity.com/paper/2002.06012/) (2020)
- [HarperValleyBank: A Domain-Specific Spoken Dialog Corpus](https://mlsanity.com/paper/2010.13929/) (2020)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [A Conformer-based ASR Frontend for Joint Acoustic Echo Cancellation, Speech Enhancement and Speech Separation](https://mlsanity.com/paper/2111.09935/) (2021)
- [Speak or Chat with Me: End-to-End Spoken Language Understanding System with Flexible Inputs](https://mlsanity.com/paper/2104.05752/) (2021)
- [Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding](https://mlsanity.com/paper/2204.05169/) (2022)
- [Two-Pass Low Latency End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.06670/) (2022)
- [A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks](https://mlsanity.com/paper/1805.06280/) (2018)

## Built on by (most cited first, 3 of 3)

- [ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems](https://mlsanity.com/paper/2503.08533/) (2025)
- [How to Enhance Causal Discrimination of Utterances: A Case on Affective Reasoning](https://mlsanity.com/paper/2305.02615/) (2023)
- [Promptformer: Prompted Conformer Transducer for ASR](https://mlsanity.com/paper/2401.07360/) (2024)

## Abstract (arXiv)

Most human interactions occur in the form of spoken conversations where the semantic meaning of a given utterance depends on the context. Each utterance in spoken conversation can be represented by many semantic and speaker attributes, and there has been an interest in building Spoken Language Understanding (SLU) systems for automatically predicting these attributes. Recent work has shown that incorporating dialogue history can help advance SLU performance. However, separate models are used for each SLU task, leading to an increase in inference time and computation cost. Motivated by this, we aim to ask: can we jointly model all the SLU tasks while incorporating context to facilitate low-latency and lightweight inference? To answer this, we propose a novel model architecture that learns dialog context to jointly predict the intent, dialog act, speaker role, and emotion for the spoken utterance. Note that our joint prediction is based on an autoregressive model and we need to decide the prediction order of dialog attributes, which is not trivial. To mitigate the issue, we also propose an order agnostic training method. Our experiments show that our joint model achieves similar results to task-specific classifiers and can effectively integrate dialog context to further improve the SLU performance.