# HAN: Higher-order Attention Network for Spoken Language Understanding

arXiv: https://arxiv.org/abs/2108.11916 · HTML: https://mlsanity.com/paper/2108.11916/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1907.00390/) (2019)
- [Continuously Differentiable Exponential Linear Units](https://mlsanity.com/paper/1704.07483/) (2017)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding](https://mlsanity.com/paper/1909.06937/) (2019)
- [Hierarchically-Refined Label Attention Network for Sequence Labeling](https://mlsanity.com/paper/1908.08676/) (2019)
- [PIN: A Novel Parallel Interactive Network for Spoken Language Understanding](https://mlsanity.com/paper/2009.13431/) (2020)
- [Injecting Word Information with Multi-Level Word Adapter for Chinese Spoken Language Understanding](https://mlsanity.com/paper/2010.03903/) (2020)

## Abstract (arXiv)

Spoken Language Understanding (SLU), including intent detection and slot filling, is a core component in human-computer interaction. The natural attributes of the relationship among the two subtasks make higher requirements on fine-grained feature interaction, i.e., the token-level intent features and slot features. Previous works mainly focus on jointly modeling the relationship between the two subtasks with attention-based models, while ignoring the exploration of attention order. In this paper, we propose to replace the conventional attention with our proposed Bilinear attention block and show that the introduced Higher-order Attention Network (HAN) brings improvement for the SLU task. Importantly, we conduct wide analysis to explore the effectiveness brought from the higher-order attention.