# Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models

arXiv: https://arxiv.org/abs/1910.09463 · HTML: https://mlsanity.com/paper/1910.09463/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2019.

## Built on by (most cited first, 24 of 24)

- [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021)
- [Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2104.07253/) (2021)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [End-to-End Spoken Language Understanding for Generalized Voice Assistants](https://mlsanity.com/paper/2106.09009/) (2021)
- [Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification](https://mlsanity.com/paper/2102.07370/) (2021)
- [Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers](https://mlsanity.com/paper/2104.01604/) (2021)
- [Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding](https://mlsanity.com/paper/2106.15065/) (2021)
- [End-to-end model for named entity recognition from speech without paired training data](https://mlsanity.com/paper/2204.00803/) (2022)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding](https://mlsanity.com/paper/2210.05291/) (2022)
- [Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2112.06743/) (2021)
- [Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training](https://mlsanity.com/paper/2305.12793/) (2023)
- [On the Transferability of Whisper-based Representations for "In-the-Wild" Cross-Task Downstream Speech Applications](https://mlsanity.com/paper/2305.14546/) (2023)
- [End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting](https://mlsanity.com/paper/2207.08179/) (2022)
- [Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text models](https://mlsanity.com/paper/2206.14589/) (2022)
- [Semantic enrichment towards efficient speech representations](https://mlsanity.com/paper/2307.01323/) (2023)
- [A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding](https://mlsanity.com/paper/2406.12141/) (2024)
- [Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification](https://mlsanity.com/paper/2109.13486/) (2021)
- [A Study of Different Ways to Use The Conformer Model For Spoken Language Understanding](https://mlsanity.com/paper/2204.03879/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)
- [Jaco: An Offline Running Privacy-aware Voice Assistant](https://mlsanity.com/paper/2209.07775/) (2022)
- [GRASS: Unified Generation Model for Speech-to-Semantic Tasks](https://mlsanity.com/paper/2309.02780/) (2023)
- [Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis](https://mlsanity.com/paper/2310.05374/) (2023)
- [Creating Spoken Dialog Systems in Ultra-Low Resourced Settings](https://mlsanity.com/paper/2312.06266/) (2023)

## Abstract (arXiv)

End-to-end models are an attractive new approach to spoken language understanding (SLU) in which the meaning of an utterance is inferred directly from the raw audio without employing the standard pipeline composed of a separately trained speech recognizer and natural language understanding module. The downside of end-to-end SLU is that in-domain speech data must be recorded to train the model. In this paper, we propose a strategy for overcoming this requirement in which speech synthesis is used to generate a large synthetic training dataset from several artificial speakers. Experiments on two open-source SLU datasets confirm the effectiveness of our approach, both as a sole source of training data and as a form of data augmentation.