# End-to-End Spoken Language Understanding Without Full Transcripts

arXiv: https://arxiv.org/abs/2009.14386 · HTML: https://mlsanity.com/paper/2009.14386/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [End-to-End Spoken Language Understanding for Generalized Voice Assistants](https://mlsanity.com/paper/2106.09009/) (2021)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding](https://mlsanity.com/paper/2106.15065/) (2021)
- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [A Streaming End-to-End Framework For Spoken Language Understanding](https://mlsanity.com/paper/2105.10042/) (2021)
- [FANS: Fusing ASR and NLU for on-device SLU](https://mlsanity.com/paper/2111.00400/) (2021)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)
- [Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target](https://mlsanity.com/paper/2305.18096/) (2023)
- [Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2204.05188/) (2022)
- [End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English](https://mlsanity.com/paper/2211.07710/) (2022)
- [SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models](https://mlsanity.com/paper/2507.19361/) (2025)
- [Improving End-to-End Models for Set Prediction in Spoken Language Understanding](https://mlsanity.com/paper/2201.12105/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)
- [End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoders](https://mlsanity.com/paper/2305.02937/) (2023)

## Abstract (arXiv)

An essential component of spoken language understanding (SLU) is slot filling: representing the meaning of a spoken utterance using semantic entity labels. In this paper, we develop end-to-end (E2E) spoken language understanding systems that directly convert speech input to semantic entities and investigate if these E2E SLU models can be trained solely on semantic entity annotations without word-for-word transcripts. Training such models is very useful as they can drastically reduce the cost of data collection. We created two types of such speech-to-entities models, a CTC model and an attention-based encoder-decoder model, by adapting models trained originally for speech recognition. Given that our experiments involve speech input, these systems need to recognize both the entity label and words representing the entity value correctly. For our speech-to-entities experiments on the ATIS corpus, both the CTC and attention models showed impressive ability to skip non-entity words: there was little degradation when trained on just entities versus full transcripts. We also explored the scenario where the entities are in an order not necessarily related to spoken order in the utterance. With its ability to do re-ordering, the attention model did remarkably well, achieving only about 2% degradation in speech-to-bag-of-entities F1 score.