# Capture Salient Historical Information: A Fast and Accurate Non-Autoregressive Model for Multi-turn Spoken Language Understanding

arXiv: https://arxiv.org/abs/2206.12209 · HTML: https://mlsanity.com/paper/2206.12209/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 56 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](https://mlsanity.com/paper/1705.05414/) (2017)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1907.00390/) (2019)
- [Joint Slot Filling and Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1812.09471/) (2018)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling](https://mlsanity.com/paper/1812.10235/) (2018)
- [Legal Judgment Prediction via Multi-Perspective Bi-Feedback Network](https://mlsanity.com/paper/1905.03969/) (2019)
- [SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2010.02693/) (2020)
- [Towards Zero-Shot Frame Semantic Parsing for Domain Scaling](https://mlsanity.com/paper/1707.02363/) (2017)
- [CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding](https://mlsanity.com/paper/1909.06937/) (2019)
- [GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2106.01925/) (2021)
- [A Study of Non-autoregressive Model for Sequence Generation](https://mlsanity.com/paper/2004.10454/) (2020)
- [Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling](https://mlsanity.com/paper/1601.01530/) (2016)
- [Learning Light-Weight Translation Models from Deep Transformer](https://mlsanity.com/paper/2012.13866/) (2020)
- [Dialogue History Matters! Personalized Response Selectionin Multi-turn Retrieval-based Chatbots](https://mlsanity.com/paper/2103.09534/) (2021)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation](https://mlsanity.com/paper/2104.02308/) (2021)
- [Sequential Dialogue Context Modeling for Spoken Language Understanding](https://mlsanity.com/paper/1705.03455/) (2017)

## Built on by (most cited first, 2 of 2)

- [A Scope Sensitive and Result Attentive Model for Multi-Intent Spoken Language Understanding](https://mlsanity.com/paper/2211.12220/) (2022)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)

## Abstract (arXiv)

Spoken Language Understanding (SLU), a core component of the task-oriented dialogue system, expects a shorter inference facing the impatience of human users. Existing work increases inference speed by designing non-autoregressive models for single-turn SLU tasks but fails to apply to multi-turn SLU in confronting the dialogue history. The intuitive idea is to concatenate all historical utterances and utilize the non-autoregressive models directly. However, this approach seriously misses the salient historical information and suffers from the uncoordinated-slot problems. To overcome those shortcomings, we propose a novel model for multi-turn SLU named Salient History Attention with Layer-Refined Transformer (SHA-LRT), which composes of an SHA module, a Layer-Refined Mechanism (LRM), and a Slot Label Generation (SLG) task. SHA captures salient historical information for the current dialogue from both historical utterances and results via a well-designed history-attention mechanism. LRM predicts preliminary SLU results from Transformer's middle states and utilizes them to guide the final prediction, and SLG obtains the sequential dependency information for the non-autoregressive encoder. Experiments on public datasets indicate that our model significantly improves multi-turn SLU performance (17.5% on Overall) with accelerating (nearly 15 times) the inference process over the state-of-the-art baseline as well as effective on the single-turn SLU tasks.