mlsanityData to 7 Oct 2026

On joint training with interfaces for spoken language understanding

arXiv 2106.15919 ↗First posted 30 Jun 2021cs.CL cs.SD eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Spoken language understanding (SLU) systems extract both text transcripts and semantics associated with intents and slots from input speech utterances. SLU systems usually consist of (1) an automatic speech recognition (ASR) module, (2) an interface module that exposes relevant outputs from ASR, and (3) a natural language understanding (NLU) module. Interfaces in SLU systems carry information on text transcriptions or richer information like neural embeddings from ASR to NLU. In this paper, we study how interfaces affect joint-training for spoken language understanding. Most notably, we obtain the state-of-the-art results on the publicly available 50-hr SLURP dataset. We first leverage large-size pretrained ASR and NLU models that are connected by a text interface, and then jointly train both models via a sequence loss function. For scenarios where pretrained models are not utilized, the best results are obtained through a joint sequence loss training using richer neural interfaces. Finally, we show the overall diminishing impact of leveraging pretrained models with increased training data size.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

23 of 31 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  6. 2015Listen, Attend and Spellcited by 542
  7. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  8. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  9. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  10. 2019BERT for Joint Intent Classification and Slot Fillingcited by 195
  11. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  12. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
11 more
  1. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  2. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
  3. 2020Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognitioncited by 34
  4. 2021Speech-language Pre-training for End-to-end Spoken Language Understandingcited by 27
  5. 2020Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfacescited by 24
  6. 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understandingcited by 22
  7. 2020Exploring Transfer Learning For End-to-End Spoken Language Understandingcited by 17
  8. 2020Adapting Pretrained Transformer to Lattices for Spoken Language Understandingcited by 12
  9. 2021Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understandingcited by 9
  10. 2021Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-Endcited by 8
  11. 2021RNN Transducer Models For Spoken Language Understandingcited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 1.

5 papers.

  1. 2022End-to-end Spoken Language Understanding with Tree-constrained Pointer Generatorcited by 6
  2. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  3. 2023CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Trainingcited by 3
  4. 2023End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoderscited by 0
  5. 2025Joint Automatic Speech Recognition And Structure Learning For Better Speech Understandingcited by 0