# Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target

arXiv: https://arxiv.org/abs/2305.18096 · HTML: https://mlsanity.com/paper/2305.18096/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 27 of 27)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining](https://mlsanity.com/paper/2010.13826/) (2020)
- [textless-lib: a Library for Textless Spoken Language Processing](https://mlsanity.com/paper/2202.07359/) (2022)
- [STOP: A dataset for Spoken Task Oriented Semantic Parsing](https://mlsanity.com/paper/2207.10643/) (2022)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [End-to-End Spoken Language Understanding Without Full Transcripts](https://mlsanity.com/paper/2009.14386/) (2020)
- [End-to-End Spoken Language Understanding for Generalized Voice Assistants](https://mlsanity.com/paper/2106.09009/) (2021)
- [DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering](https://mlsanity.com/paper/2203.04911/) (2022)
- [Analyzing the Robustness of Unsupervised Speech Recognition](https://mlsanity.com/paper/2110.03509/) (2021)
- [End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features](https://mlsanity.com/paper/2011.08238/) (2020)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)

## Built on by (most cited first, 6 of 6)

- [Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2411.01834/) (2024)
- [LAST: Language Model Aware Speech Tokenization](https://mlsanity.com/paper/2409.03701/) (2024)
- [Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERT](https://mlsanity.com/paper/2409.10103/) (2024)
- [Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model](https://mlsanity.com/paper/2402.05819/) (2024)
- [HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model](https://mlsanity.com/paper/2310.03975/) (2023)
- [Representing Speech Through Autoregressive Prediction of Cochlear Tokens](https://mlsanity.com/paper/2508.11598/) (2025)

## Abstract (arXiv)

Spoken Language Understanding (SLU) is a task that aims to extract semantic information from spoken utterances. Previous research has made progress in end-to-end SLU by using paired speech-text data, such as pre-trained Automatic Speech Recognition (ASR) models or paired text as intermediate targets. However, acquiring paired transcripts is expensive and impractical for unwritten languages. On the other hand, Textless SLU extracts semantic information from speech without utilizing paired transcripts. However, the absence of intermediate targets and training guidance for textless SLU often results in suboptimal performance. In this work, inspired by the content-disentangled discrete units from self-supervised speech models, we proposed to use discrete units as intermediate guidance to improve textless SLU performance. Our method surpasses the baseline method on five SLU benchmark corpora. Additionally, we find that unit guidance facilitates few-shot learning and enhances the model's ability to handle noise.