# Sequential End-to-End Intent and Slot Label Classification and Localization

arXiv: https://arxiv.org/abs/2106.04660 · HTML: https://mlsanity.com/paper/2106.04660/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 13 of 13)

- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks](https://mlsanity.com/paper/1609.01462/) (2016)
- [Improving End-to-End Speech-to-Intent Classification with Reptile](https://mlsanity.com/paper/2008.01994/) (2020)
- [A low latency ASR-free end to end spoken language understanding system](https://mlsanity.com/paper/2011.04884/) (2020)
- [RNN based Incremental Online Spoken Language Understanding](https://mlsanity.com/paper/1910.10287/) (2019)
- [Learning spectro-temporal features with 3D CNNs for speech emotion recognition](https://mlsanity.com/paper/1708.05071/) (2017)
- [Connectionist Temporal Localization for Sound Event Detection with Sequential Labeling](https://mlsanity.com/paper/1810.09052/) (2018)

## Built on by (most cited first, 7 of 7)

- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.14318/) (2022)
- [Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text models](https://mlsanity.com/paper/2206.14589/) (2022)
- [Device Directedness with Contextual Cues for Spoken Dialog Systems](https://mlsanity.com/paper/2211.13280/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)
- [Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding](https://mlsanity.com/paper/2405.15097/) (2024)
- [S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling](https://mlsanity.com/paper/2602.11065/) (2026)

## Abstract (arXiv)

Human-computer interaction (HCI) is significantly impacted by delayed responses from a spoken dialogue system. Hence, end-to-end (e2e) spoken language understanding (SLU) solutions have recently been proposed to decrease latency. Such approaches allow for the extraction of semantic information directly from the speech signal, thus bypassing the need for a transcript from an automatic speech recognition (ASR) system. In this paper, we propose a compact e2e SLU architecture for streaming scenarios, where chunks of the speech signal are processed continuously to predict intent and slot values. Our model is based on a 3D convolutional neural network (3D-CNN) and a unidirectional long short-term memory (LSTM). We compare the performance of two alignment-free losses: the connectionist temporal classification (CTC) method and its adapted version, namely connectionist temporal localization (CTL). The latter performs not only the classification but also localization of sequential audio events. The proposed solution is evaluated on the Fluent Speech Command dataset and results show our model ability to process incoming speech signal, reaching accuracy as high as 98.97 % for CTC and 98.78 % for CTL on single-label classification, and as high as 95.69 % for CTC and 95.28 % for CTL on two-label prediction.