# End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features

arXiv: https://arxiv.org/abs/2011.08238 · HTML: https://mlsanity.com/paper/2011.08238/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 8 of 8)

- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [FANS: Fusing ASR and NLU for on-device SLU](https://mlsanity.com/paper/2111.00400/) (2021)
- [Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding](https://mlsanity.com/paper/2204.05169/) (2022)
- [Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target](https://mlsanity.com/paper/2305.18096/) (2023)
- [End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English](https://mlsanity.com/paper/2211.07710/) (2022)
- [Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understanding](https://mlsanity.com/paper/2307.11005/) (2023)
- [Enhancing Speech Recognition Decoding via Layer Aggregation](https://mlsanity.com/paper/2203.11325/) (2022)
- [Improving End-to-End Models for Set Prediction in Spoken Language Understanding](https://mlsanity.com/paper/2201.12105/) (2022)

## Abstract (arXiv)

Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of spoken language understanding (SLU) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre-trained acoustic features, pre-trained model initialization and multi-task training. Several SLU experiments for predicting intent and entity labels/values using the ATIS dataset are performed. These experiments investigate the interaction of pre-trained model initialization and multi-task training with either traditional filterbank or self-supervised pre-trained acoustic features. Results show not only that self-supervised pre-trained acoustic features outperform filterbank features in almost all the experiments, but also that when these features are used in combination with multi-task training, they almost eliminate the necessity of pre-trained model initialization.