mlsanityData to 14 Oct 2026

A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding

arXiv 2406.12141 ↗First posted 17 Jun 2024cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning is vastly used to efficiently represent speech for Spoken Language Understanding, gradually replacing conventional approaches. Meanwhile, textual SSL models are proposed to encode language-agnostic semantics. SAMU-XLSR framework employed this semantic information to enrich multilingual speech representations. A recent study investigated SAMU-XLSR in-domain semantic enrichment by specializing it on downstream transcriptions, leading to state-of-the-art results on a challenging SLU task. This study's interest lies in the loss of multilingual performances and lack of specific-semantics training induced by such specialization in close languages without any SLU implication. We also consider SAMU-XLSR's loss of initial cross-lingual abilities due to a separate SLU fine-tuning. Therefore, this paper proposes a dual task learning approach to improve SAMU-XLSR semantic enrichment while considering distant languages for multilingual and language portability experiments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

22 of 35 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  7. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  8. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  9. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  10. 2018Towards end-to-end spoken language understandingcited by 92
  11. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  12. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
10 more
  1. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  2. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  3. 2019Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portabilitycited by 29
  4. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  5. 2019Using Speech Synthesis to Train End-to-End Spoken Language Understanding Modelscited by 24
  6. 2020On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognitioncited by 19
  7. 2021Where are we in semantic concept extraction for Spoken Language Understanding?cited by 11
  8. 2022Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Modelscited by 10
  9. 2022On the Use of Semantically-Aligned Speech Representations for Spoken Language Understandingcited by 7
  10. 2023Semantic enrichment towards efficient speech representationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unified Vision-Language Modeling via Concept Space Alignmentcited by 0