# Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding arXiv: https://arxiv.org/abs/2106.15065 ยท HTML: https://mlsanity.com/paper/2106.15065/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 36 references that are in the map. - First posted on arXiv 29 Jun 2021. ## Builds on (most cited first, 24 of 24) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016) - [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018) - [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015) - [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017) - [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019) - [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020) - [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017) - [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019) - [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019) - [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020) - [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020) - [Pre-training on high-resource speech recognition improves low-resource speech-to-text translation](https://mlsanity.com/paper/1809.01431/) (2018) - [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020) - [Transfer Learning for Speech and Language Processing](https://mlsanity.com/paper/1511.06066/) (2015) - [Spoken Language Understanding on the Edge](https://mlsanity.com/paper/1810.12735/) (2018) - [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021) - [Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models](https://mlsanity.com/paper/1910.09463/) (2019) - [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021) - [Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2011.09044/) (2020) - [End-to-End Spoken Language Understanding Without Full Transcripts](https://mlsanity.com/paper/2009.14386/) (2020) - [Semantic Complexity in End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2008.02858/) (2020) - [Streaming Models for Joint Speech Recognition and Translation](https://mlsanity.com/paper/2101.09149/) (2021) - [A Data Efficient End-To-End Spoken Language Understanding Architecture](https://mlsanity.com/paper/2002.05955/) (2020) ## Built on by (most cited first, 11 of 11) - [Self-Supervised Speech Representations are More Phonetic than Semantic](https://mlsanity.com/paper/2406.08619/) (2024) - [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021) - [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021) - [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022) - [Two-Pass Low Latency End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.06670/) (2022) - [Multitask Learning for Low Resource Spoken Language Understanding](https://mlsanity.com/paper/2211.13703/) (2022) - [Follow-on Question Suggestion via Voice Hints for Voice Assistants](https://mlsanity.com/paper/2310.17034/) (2023) - [Out-of-distribution generalisation in spoken language understanding](https://mlsanity.com/paper/2407.07425/) (2024) - [A Study of Different Ways to Use The Conformer Model For Spoken Language Understanding](https://mlsanity.com/paper/2204.03879/) (2022) - [Bidirectional Representations for Low Resource Spoken Language Understanding](https://mlsanity.com/paper/2211.14320/) (2022) - [Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations](https://mlsanity.com/paper/2402.01298/) (2024) ## Abstract (arXiv) Decomposable tasks are complex and comprise of a hierarchy of sub-tasks. Spoken intent prediction, for example, combines automatic speech recognition and natural language understanding. Existing benchmarks, however, typically hold out examples for only the surface-level sub-task. As a result, models with similar performance on these benchmarks may have unobserved performance differences on the other sub-tasks. To allow insightful comparisons between competitive end-to-end architectures, we propose a framework to construct robust test sets using coordinate ascent over sub-task specific utility functions. Given a dataset for a decomposable task, our method optimally creates a test set for each sub-task to individually assess sub-components of the end-to-end model. Using spoken language understanding as a case study, we generate new splits for the Fluent Speech Commands and Snips SmartLights datasets. Each split has two test sets: one with held-out utterances assessing natural language understanding abilities, and one with held-out speakers to test speech processing skills. Our splits identify performance gaps up to 10% between end-to-end systems that were within 1% of each other on the original test sets. These performance gaps allow more realistic and actionable comparisons between different architectures, driving future model development. We release our splits and tools for the community.