mlsanityData to 7 Oct 2026

Speech-language Pre-training for End-to-end Spoken Language Understanding

arXiv 2102.06283 ↗First posted 11 Feb 2021cs.CL cs.SD eess.AScited by 27

Not on the map. Why?

Abstract (arXiv)

End-to-end (E2E) spoken language understanding (SLU) can infer semantics directly from speech signal without cascading an automatic speech recognizer (ASR) with a natural language understanding (NLU) module. However, paired utterance recordings and corresponding semantics may not always be available or sufficient to train an E2E SLU model in a real production environment. In this paper, we propose to unify a well-optimized E2E ASR encoder (speech) and a pre-trained language model encoder (language) into a transformer decoder. The unified speech-language pre-trained model (SLP) is continually enhanced on limited labeled data from a target domain by using a conditional masked language model (MLM) objective, and thus can effectively generate a sequence of intent, slot type, and slot value for given input speech in the inference. The experimental results on two public corpora show that our approach to E2E SLU is superior to the conventional cascaded method. It also outperforms the present state-of-the-art approaches to E2E SLU with much less paired data.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

20 of 28 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2016Layer Normalizationcited by 5,579
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  8. 2015Attention-Based Models for Speech Recognitioncited by 400
  9. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  10. 2019BERT for Joint Intent Classification and Slot Fillingcited by 195
  11. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  12. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
8 more
  1. 2018Towards end-to-end spoken language understandingcited by 92
  2. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  3. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
  4. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  5. 2019Using Speech Synthesis to Train End-to-End Spoken Language Understanding Modelscited by 24
  6. 2020Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learningcited by 14
  7. 2020Learning Spoken Language Representations with Neural Lattice Language Modelingcited by 6
  8. 2020SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understandingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 3; 2024: 3; 2025: 3; 2026: 1.

27 papers.

Sort:

  1. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  2. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  3. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  4. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  5. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  6. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  7. 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understandingcited by 22
  8. 2021Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understandingcited by 11
  9. 2021Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learningcited by 6
  10. 2021Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speechcited by 6
  11. 2021On joint training with interfaces for spoken language understandingcited by 5
  12. 2021Do We Still Need Automatic Speech Recognition for Spoken Language Understanding?cited by 5
15 more
  1. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  2. 2022Toward Low-Cost End-to-End Spoken Language Understandingcited by 3
  3. 2022Introducing Semantics into Speech Encoderscited by 3
  4. 2022Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systemscited by 2
  5. 2022Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text modelscited by 2
  6. 2022End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and Englishcited by 2
  7. 2021Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classificationcited by 0
  8. 2021Intent Classification Using Pre-trained Language Agnostic Embeddings For Low Resource Languagescited by 0
  9. 2022A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognitioncited by 0
  10. 2023GRASS: Unified Generation Model for Speech-to-Semantic Taskscited by 0
  11. 2023Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesiscited by 0
  12. 2024Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understandingcited by 0
  13. 2025Audio-FLAN: An Instruction-Following Dataset for Unified Audio Understanding and Generation of Speech, Music, and Soundcited by 0
  14. 2025RealTalk-CN: A Realistic Chinese Speech-Text Dialogue Benchmark With Cross-Modal Interaction Analysiscited by 0
  15. 2026End-to-End Voice Intent Recognition for Spontaneous Human-Drone Interaction with Naive Userscited by 0