# FANS: Fusing ASR and NLU for on-device SLU

arXiv: https://arxiv.org/abs/2111.00400 · HTML: https://mlsanity.com/paper/2111.00400/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 31 Oct 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining](https://mlsanity.com/paper/2010.13826/) (2020)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [End-to-End Spoken Language Understanding Without Full Transcripts](https://mlsanity.com/paper/2009.14386/) (2020)
- [Improving End-to-End Speech-to-Intent Classification with Reptile](https://mlsanity.com/paper/2008.01994/) (2020)
- [Recent Advances in End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1909.13332/) (2019)
- [Adapting Pretrained Transformer to Lattices for Spoken Language Understanding](https://mlsanity.com/paper/2011.00780/) (2020)
- [Capsule Networks for Low Resource Spoken Language Understanding](https://mlsanity.com/paper/1805.02922/) (2018)
- [Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems](https://mlsanity.com/paper/2002.06012/) (2020)
- [A Data Efficient End-To-End Spoken Language Understanding Architecture](https://mlsanity.com/paper/2002.05955/) (2020)
- [End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features](https://mlsanity.com/paper/2011.08238/) (2020)
- [Topic Identification for Speech without ASR](https://mlsanity.com/paper/1703.07476/) (2017)
- [Transfer Learning for Context-Aware Spoken Language Understanding](https://mlsanity.com/paper/2003.01305/) (2020)

## Built on by (most cited first, 8 of 8)

- [Deliberation Model for On-Device Spoken Language Understanding](https://mlsanity.com/paper/2204.01893/) (2022)
- [Multi-task RNN-T with Semantic Decoder for Streamable Spoken Language Understanding](https://mlsanity.com/paper/2204.00558/) (2022)
- [Speech Understanding on Tiny Devices with A Learning Cache](https://mlsanity.com/paper/2311.18188/) (2023)
- [Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text models](https://mlsanity.com/paper/2206.14589/) (2022)
- [Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2307.12134/) (2023)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)
- [End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoders](https://mlsanity.com/paper/2305.02937/) (2023)
- [PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2406.07823/) (2024)

## Abstract (arXiv)

Spoken language understanding (SLU) systems translate voice input commands to semantics which are encoded as an intent and pairs of slot tags and values. Most current SLU systems deploy a cascade of two neural models where the first one maps the input audio to a transcript (ASR) and the second predicts the intent and slots from the transcript (NLU). In this paper, we introduce FANS, a new end-to-end SLU model that fuses an ASR audio encoder to a multi-task NLU decoder to infer the intent, slot tags, and slot values directly from a given input audio, obviating the need for transcription. FANS consists of a shared audio encoder and three decoders, two of which are seq-to-seq decoders that predict non null slot tags and slot values in parallel and in an auto-regressive manner. FANS neural encoder and decoders architectures are flexible which allows us to leverage different combinations of LSTM, self-attention, and attenders. Our experiments show compared to the state-of-the-art end-to-end SLU models, FANS reduces ICER and IRER errors relatively by 30 % and 7 %, respectively, when tested on an in-house SLU dataset and by 0.86 % and 2 % absolute when tested on a public SLU dataset.