# End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting arXiv: https://arxiv.org/abs/2207.08179 ยท HTML: https://mlsanity.com/paper/2207.08179/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 75 references that are in the map. - First posted on arXiv 17 Jul 2022. ## Builds on (most cited first, 24 of 24) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017) - [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015) - [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014) - [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017) - [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019) - [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016) - [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018) - [Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding](https://mlsanity.com/paper/1807.01554/) (2018) - [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018) - [Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability](https://mlsanity.com/paper/1906.07601/) (2019) - [Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.02720/) (2017) - [Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models](https://mlsanity.com/paper/1910.09463/) (2019) - [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020) - [Training Neural Speech Recognition Systems with Synthetic Speech Augmentation](https://mlsanity.com/paper/1811.00707/) (2018) - [Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning](https://mlsanity.com/paper/2007.01836/) (2020) - [Parallel training of DNNs with Natural Gradient and Parameter Averaging](https://mlsanity.com/paper/1410.7455/) (2014) - [Sequential Dialogue Context Modeling for Spoken Language Understanding](https://mlsanity.com/paper/1705.03455/) (2017) - [End-to-end named entity extraction from speech](https://mlsanity.com/paper/1805.12045/) (2018) - [Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding](https://mlsanity.com/paper/2005.11640/) (2020) - [Improving Slot Filling Performance with Attentive Neural Networks on Dependency Structures](https://mlsanity.com/paper/1707.01075/) (2017) - [ASR error management for improving spoken language understanding](https://mlsanity.com/paper/1705.09515/) (2017) ## Built on by (most cited first, 3 of 3) - [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023) - [Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2307.12134/) (2023) - [Spoken Dialogue System for Medical Prescription Acquisition on Smartphone: Development, Corpus and Evaluation](https://mlsanity.com/paper/2311.03510/) (2023) ## Abstract (arXiv) Spoken Language Understanding (SLU) is a core task in most human-machine interaction systems. With the emergence of smart homes, smart phones and smart speakers, SLU has become a key technology for the industry. In a classical SLU approach, an Automatic Speech Recognition (ASR) module transcribes the speech signal into a textual representation from which a Natural Language Understanding (NLU) module extracts semantic information. Recently End-to-End SLU (E2E SLU) based on Deep Neural Networks has gained momentum since it benefits from the joint optimization of the ASR and the NLU parts, hence limiting the cascade of error effect of the pipeline architecture. However, little is known about the actual linguistic properties used by E2E models to predict concepts and intents from speech input. In this paper, we present a study identifying the signal features and other linguistic properties used by an E2E model to perform the SLU task. The study is carried out in the application domain of a smart home that has to handle non-English (here French) voice commands. The results show that a good E2E SLU performance does not always require a perfect ASR capability. Furthermore, the results show the superior capabilities of the E2E model in handling background noise and syntactic variation compared to the pipeline model. Finally, a finer-grained analysis suggests that the E2E model uses the pitch information of the input signal to identify voice command concepts. The results and methodology outlined in this paper provide a springboard for further analyses of E2E models in speech processing.