# On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding

arXiv: https://arxiv.org/abs/2210.05291 · HTML: https://mlsanity.com/paper/2210.05291/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2010.04284/) (2020)
- [SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation](https://mlsanity.com/paper/2205.08180/) (2022)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability](https://mlsanity.com/paper/1906.07601/) (2019)
- [Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?](https://mlsanity.com/paper/2204.12765/) (2022)
- [Using Speech Synthesis to Train End-to-End Spoken Language Understanding Models](https://mlsanity.com/paper/1910.09463/) (2019)
- [Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2011.09044/) (2020)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [End-to-end architectures for ASR-free spoken language understanding](https://mlsanity.com/paper/1910.10599/) (2019)
- [Where are we in semantic concept extraction for Spoken Language Understanding?](https://mlsanity.com/paper/2106.13045/) (2021)
- [A Data Efficient End-To-End Spoken Language Understanding Architecture](https://mlsanity.com/paper/2002.05955/) (2020)
- [Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training](https://mlsanity.com/paper/2203.00648/) (2022)

## Built on by (most cited first, 7 of 7)

- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models](https://mlsanity.com/paper/2308.10390/) (2023)
- [SENSE models: an open source solution for multilingual and multimodal semantic-based tasks](https://mlsanity.com/paper/2509.12093/) (2025)
- [Improved Cross-Lingual Transfer Learning For Automatic Speech Translation](https://mlsanity.com/paper/2306.00789/) (2023)
- [Semantic enrichment towards efficient speech representations](https://mlsanity.com/paper/2307.01323/) (2023)
- [Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2310.06103/) (2023)
- [A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding](https://mlsanity.com/paper/2406.12141/) (2024)

## Abstract (arXiv)

In this paper we examine the use of semantically-aligned speech representations for end-to-end spoken language understanding (SLU). We employ the recently-introduced SAMU-XLSR model, which is designed to generate a single embedding that captures the semantics at the utterance level, semantically aligned across different languages. This model combines the acoustic frame-level speech representation learning model (XLS-R) with the Language Agnostic BERT Sentence Embedding (LaBSE) model. We show that the use of the SAMU-XLSR model instead of the initial XLS-R model improves significantly the performance in the framework of end-to-end SLU. Finally, we present the benefits of using this model towards language portability in SLU.