mlsanityData to 7 Oct 2026

FANS: Fusing ASR and NLU for on-device SLU

arXiv 2111.00400 ↗First posted 31 Oct 2021cs.CL cs.SD eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

Spoken language understanding (SLU) systems translate voice input commands to semantics which are encoded as an intent and pairs of slot tags and values. Most current SLU systems deploy a cascade of two neural models where the first one maps the input audio to a transcript (ASR) and the second predicts the intent and slots from the transcript (NLU). In this paper, we introduce FANS, a new end-to-end SLU model that fuses an ASR audio encoder to a multi-task NLU decoder to infer the intent, slot tags, and slot values directly from a given input audio, obviating the need for transcription. FANS consists of a shared audio encoder and three decoders, two of which are seq-to-seq decoders that predict non null slot tags and slot values in parallel and in an auto-regressive manner. FANS neural encoder and decoders architectures are flexible which allows us to leverage different combinations of LSTM, self-attention, and attenders. Our experiments show compared to the state-of-the-art end-to-end SLU models, FANS reduces ICER and IRER errors relatively by 30 % and 7 %, respectively, when tested on an in-house SLU dataset and by 0.86 % and 2 % absolute when tested on a public SLU dataset.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 31 Oct 2021.

Builds on

21 of 33 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  4. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  5. 2018Speaker Recognition from Raw Waveform with SincNetcited by 166
  6. 2018Towards end-to-end spoken language understandingcited by 92
  7. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
  8. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  9. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  10. 2020End-to-End Neural Transformer Based Spoken Language Understandingcited by 31
  11. 2020Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfacescited by 24
  12. 2020End-to-End Spoken Language Understanding Without Full Transcriptscited by 19
9 more
  1. 2020Improving End-to-End Speech-to-Intent Classification with Reptilecited by 17
  2. 2019Recent Advances in End-to-End Spoken Language Understandingcited by 13
  3. 2020Adapting Pretrained Transformer to Lattices for Spoken Language Understandingcited by 12
  4. 2018Capsule Networks for Low Resource Spoken Language Understandingcited by 11
  5. 2020Dialogue history integration into end-to-end signal-to-concept spoken language understanding systemscited by 10
  6. 2020A Data Efficient End-To-End Spoken Language Understanding Architecturecited by 8
  7. 2020End-to-end spoken language understanding using transformer networks and self-supervised pre-trained featurescited by 8
  8. 2017Topic Identification for Speech without ASRcited by 2
  9. 2020Transfer Learning for Context-Aware Spoken Language Understandingcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 1.

8 papers.

  1. 2022Deliberation Model for On-Device Spoken Language Understandingcited by 8
  2. 2022Multi-task RNN-T with Semantic Decoder for Streamable Spoken Language Understandingcited by 3
  3. 2023Speech Understanding on Tiny Devices with A Learning Cachecited by 3
  4. 2022Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text modelscited by 2
  5. 2023Modality Confidence Aware Training for Robust End-to-End Spoken Language Understandingcited by 2
  6. 2022Meta Auxiliary Learning for Low-resource Spoken Language Understandingcited by 0
  7. 2023End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoderscited by 0
  8. 2024PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understandingcited by 0