mlsanityData to 14 Oct 2026

Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding

arXiv 2305.12301 ↗First posted 20 May 2023cs.CL cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

The pre-trained speech encoder wav2vec 2.0 performs very well on various spoken language understanding (SLU) tasks. However, on many tasks, it trails behind text encoders with textual input. To improve the understanding capability of SLU encoders, various studies have used knowledge distillation to transfer knowledge from natural language understanding (NLU) encoders. We use a very simple method of distilling from a textual sentence embedder directly into wav2vec 2.0 as pre-training, utilizing paired audio-text datasets. We observed that this method is indeed capable of improving SLU task performance in fine-tuned settings, as well as full-data and few-shot transfer on a frozen encoder. However, the model performs worse on certain tasks highlighting the strengths and weaknesses of our approach.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

14 of 18 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  8. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  9. 2023A Review of Deep Learning Techniques for Speech Processingcited by 52
  10. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
  11. 2020Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learningcited by 14
  12. 2020Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillationcited by 11
2 more
  1. 2020Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understandingcited by 9
  2. 2021Do We Still Need Automatic Speech Recognition for Spoken Language Understanding?cited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Multimodal Modeling For Spoken Language Identificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model
  2. 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding
  3. 2025AFD-SLU: Adaptive Feature Distillation for Spoken Language Understanding
  4. 2023Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding
  5. 2023Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings
  6. 2023Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
  7. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
  8. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders