# Compositional Generalization in Spoken Language Understanding

arXiv: https://arxiv.org/abs/2312.15815 · HTML: https://mlsanity.com/paper/2312.15815/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [Compositional generalization through meta sequence-to-sequence learning](https://mlsanity.com/paper/1906.05381/) (2019)
- [Compositional Generalization via Neural-Symbolic Stack Machines](https://mlsanity.com/paper/2008.06662/) (2020)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Learning Compositional Rules via Neural Program Synthesis](https://mlsanity.com/paper/2003.05562/) (2020)
- [Improve Transformer Models with Better Relative Position Embeddings](https://mlsanity.com/paper/2009.13658/) (2020)
- [Making Transformers Solve Compositional Tasks](https://mlsanity.com/paper/2108.04378/) (2021)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling](https://mlsanity.com/paper/1812.10235/) (2018)
- [Environmental drivers of systematicity and generalization in a situated agent](https://mlsanity.com/paper/1910.00571/) (2019)
- [Compositional Networks Enable Systematic Generalization for Grounded Language Understanding](https://mlsanity.com/paper/2008.02742/) (2020)
- [Using Syntax to Ground Referring Expressions in Natural Images](https://mlsanity.com/paper/1805.10547/) (2018)
- [Systematic Generalization on gSCAN with Language Conditioned Embedding](https://mlsanity.com/paper/2009.05552/) (2020)
- [OneNet: Joint Domain, Intent, Slot Prediction for Spoken Language Understanding](https://mlsanity.com/paper/1801.05149/) (2018)
- [Think before you act: A simple baseline for compositional generalization](https://mlsanity.com/paper/2009.13962/) (2020)
- [Data Augmentation with Atomic Templates for Spoken Language Understanding](https://mlsanity.com/paper/1908.10770/) (2019)
- [Iterative Delexicalization for Improved Spoken Language Understanding](https://mlsanity.com/paper/1910.07060/) (2019)

## Built on by (most cited first, 1 of 1)

- [Out-of-distribution generalisation in spoken language understanding](https://mlsanity.com/paper/2407.07425/) (2024)

## Abstract (arXiv)

State-of-the-art spoken language understanding (SLU) models have shown tremendous success in benchmark SLU datasets, yet they still fail in many practical scenario due to the lack of model compositionality when trained on limited training data. In this paper, we study two types of compositionality: (a) novel slot combination, and (b) length generalization. We first conduct in-depth analysis, and find that state-of-the-art SLU models often learn spurious slot correlations during training, which leads to poor performance in both compositional cases. To mitigate these limitations, we create the first compositional splits of benchmark SLU datasets and we propose the first compositional SLU model, including compositional loss and paired training that tackle each compositional case respectively. On both benchmark and compositional splits in ATIS and SNIPS, we show that our compositional SLU model significantly outperforms (up to 5% F1 score) state-of-the-art BERT SLU model.