# Towards end-to-end spoken language understanding

arXiv: https://arxiv.org/abs/1802.08395 · HTML: https://mlsanity.com/paper/1802.08395/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2018.

## Built on by (most cited first, 30 of 92)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021)
- [End-to-end Spoken Conversational Question Answering: Task, Dataset and Model](https://mlsanity.com/paper/2204.14272/) (2022)
- [4D Gaussian Splatting with Scale-aware Residual Field and Adaptive Optimization for Real-time Rendering of Temporally Complex Dynamic Scenes](https://mlsanity.com/paper/2412.06299/) (2024)
- [Self-supervised Dialogue Learning for Spoken Conversational Question Answering](https://mlsanity.com/paper/2106.02182/) (2021)
- [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021)
- [Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2104.07253/) (2021)
- [STOP: A dataset for Spoken Task Oriented Semantic Parsing](https://mlsanity.com/paper/2207.10643/) (2022)
- [Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript](https://mlsanity.com/paper/2102.00804/) (2021)
- [Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding](https://mlsanity.com/paper/2305.13512/) (2023)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification](https://mlsanity.com/paper/2108.02598/) (2021)
- [LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models](https://mlsanity.com/paper/2308.10390/) (2023)
- [Speech Retrieval-Augmented Generation without Automatic Speech Recognition](https://mlsanity.com/paper/2412.16500/) (2024)
- [Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification](https://mlsanity.com/paper/2102.07370/) (2021)
- [Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers](https://mlsanity.com/paper/2104.01604/) (2021)
- [Where are we in semantic concept extraction for Spoken Language Understanding?](https://mlsanity.com/paper/2106.13045/) (2021)
- [End-to-end model for named entity recognition from speech without paired training data](https://mlsanity.com/paper/2204.00803/) (2022)
- [End2End Acoustic to Semantic Transduction](https://mlsanity.com/paper/2102.01013/) (2021)
- [Towards Relation Extraction From Speech](https://mlsanity.com/paper/2210.08759/) (2022)
- [Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding](https://mlsanity.com/paper/2102.06750/) (2021)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understanding](https://mlsanity.com/paper/2108.13048/) (2021)
- [Speak or Chat with Me: End-to-End Spoken Language Understanding System with Flexible Inputs](https://mlsanity.com/paper/2104.05752/) (2021)

## Abstract (arXiv)

Spoken language understanding system is traditionally designed as a pipeline of a number of components. First, the audio signal is processed by an automatic speech recognizer for transcription or n-best hypotheses. With the recognition results, a natural language understanding system classifies the text to structured data as domain, intent and slots for down-streaming consumers, such as dialog system, hands-free applications. These components are usually developed and optimized independently. In this paper, we present our study on an end-to-end learning system for spoken language understanding. With this unified approach, we can infer the semantic meaning directly from audio features without the intermediate text representation. This study showed that the trained model can achieve reasonable good result and demonstrated that the model can capture the semantic attention directly from the audio features.