mlsanityData to 14 Oct 2026

Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training

arXiv 2305.12793 ↗First posted 22 May 2023eess.AS cs.CL cs.MMcited by 5

Not on the map. Why?

Abstract (arXiv)

End-to-end (E2E) spoken language understanding (SLU) is constrained by the cost of collecting speech-semantics pairs, especially when label domains change. Hence, we explore textitzero-shot E2E SLU, which learns E2E SLU without speech-semantics pairs, instead using only speech-text and text-semantics pairs. Previous work achieved zero-shot by pseudolabeling all speech-text transcripts with a natural language understanding (NLU) model learned on text-semantics corpora. However, this method requires the domains of speech-text and text-semantics to match, which often mismatch due to separate collections. Furthermore, using the entire collected speech-text corpus from any domains leads to textitimbalance and textitnoise issues. To address these, we propose textitcross-modal selective self-training (CMSST). CMSST tackles imbalance by clustering in a joint space of the three modalities (speech, text, and semantics) and handles label noise with a selection network. We also introduce two benchmarks for zero-shot E2E SLU, covering matched and found speech (mismatched) settings. Experiments show that CMSST improves performance in both two settings, with significantly reduced sample sizes and training time. Our code and data are released in https://github.com/amazon-science/zero-shot-E2E-slu.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

31 of 58 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  6. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  7. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  8. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  9. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  10. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  11. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  12. 2019Self-Training for End-to-End Speech Recognitioncited by 129
19 more
  1. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  2. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  3. 2018Towards end-to-end spoken language understandingcited by 92
  4. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  5. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  6. 2021Reconsidering Representation Alignment for Multi-view Clusteringcited by 57
  7. 2021Self-Supervised Discriminative Feature Learning for Deep Multi-View Clusteringcited by 41
  8. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  9. 2022Fast Multi-view Clustering via Ensembles: Towards Scalability, Superiority, and Simplicitycited by 28
  10. 2019Using Speech Synthesis to Train End-to-End Spoken Language Understanding Modelscited by 24
  11. 2020Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfacescited by 24
  12. 2022Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systemscited by 20
  13. 2020Exploring Transfer Learning For End-to-End Spoken Language Understandingcited by 17
  14. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  15. 2020Rethink Maximum Mean Discrepancy for Domain Adaptationcited by 11
  16. 2022End-to-end model for named entity recognition from speech without paired training datacited by 11
  17. 2020Learning Not to Learn in the Presence of Noisy Labelscited by 9
  18. 2021Online Selective Classification with Limited Feedbackcited by 9
  19. 2018Fast and Scalable Expansion of Natural Language Understanding Functionality for Intelligent Agentscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023Uncertainty Estimation on Sequential Labeling via Uncertainty Transmissioncited by 10
  2. 2024Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selectioncited by 5
  3. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  4. 2023Few-Shot Spoken Language Understanding via Joint Speech-Text Modelscited by 1
  5. 2024CERET: Cost-Effective Extrinsic Refinement for Text Generationcited by 1