# Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning

arXiv: https://arxiv.org/abs/2007.01836 · HTML: https://mlsanity.com/paper/2007.01836/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021)
- [Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification](https://mlsanity.com/paper/2108.02598/) (2021)
- [Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification](https://mlsanity.com/paper/2102.07370/) (2021)
- [Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding](https://mlsanity.com/paper/2204.05169/) (2022)
- [An Analysis of Semantically-Aligned Speech-Text Embeddings](https://mlsanity.com/paper/2204.01235/) (2022)
- [Multi-task RNN-T with Semantic Decoder for Streamable Spoken Language Understanding](https://mlsanity.com/paper/2204.00558/) (2022)
- [End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting](https://mlsanity.com/paper/2207.08179/) (2022)
- [Adaptive Knowledge Distillation between Text and Speech Pre-trained Models](https://mlsanity.com/paper/2303.03600/) (2023)
- [Speech Understanding on Tiny Devices with A Learning Cache](https://mlsanity.com/paper/2311.18188/) (2023)
- [Effectiveness of Text, Acoustic, and Lattice-based representations in Spoken Language Understanding tasks](https://mlsanity.com/paper/2212.08489/) (2022)
- [Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding](https://mlsanity.com/paper/2305.12301/) (2023)
- [Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2310.06103/) (2023)
- [Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification](https://mlsanity.com/paper/2109.13486/) (2021)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)

## Abstract (arXiv)

Spoken language understanding is typically based on pipeline architectures including speech recognition and natural language understanding steps. These components are optimized independently to allow usage of available data, but the overall system suffers from error propagation. In this paper, we propose a novel training method that enables pretrained contextual embeddings to process acoustic features. In particular, we extend it with an encoder of pretrained speech recognition systems in order to construct end-to-end spoken language understanding systems. Our proposed method is based on the teacher-student framework across speech and text modalities that aligns the acoustic and the semantic latent spaces. Experimental results in three benchmarks show that our system reaches the performance comparable to the pipeline architecture without using any training data and outperforms it after fine-tuning with ten examples per class on two out of three benchmarks.