mlsanityData to 7 Oct 2026

DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding

arXiv 2406.09345 ↗First posted 13 Jun 2024cs.CL cs.SD eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

The integration of pre-trained text-based large language models (LLM) with speech input has enabled instruction-following capabilities for diverse speech tasks. This integration requires the use of a speech encoder, a speech adapter, and an LLM, trained on diverse tasks. We propose the use of discrete speech units (DSU), rather than continuous-valued speech encoder outputs, that are converted to the LLM token embedding space using the speech adapter. We generate DSU using a self-supervised speech encoder followed by k-means clustering. The proposed model shows robust performance on speech inputs from seen/unseen domains and instruction-following capability in spoken question answering. We also explore various types of DSU extracted from different layers of the self-supervised speech encoder, as well as Mel frequency Cepstral Coefficients (MFCC). Our findings suggest that the ASR task and datasets are not crucial in instruction-tuning for spoken question answering tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 34 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

33 of 34 references.

References · 33

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Mistral 7Bcited by 3,065
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  8. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  12. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
21 more
  1. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  2. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  3. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  4. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  5. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  6. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  7. 2023Joint Audio and Speech Understandingcited by 99
  8. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  9. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  10. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  11. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  12. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  13. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  14. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  15. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 34
  16. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  17. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  18. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  19. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  20. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  21. 2023Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  3. 2025QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptionscited by 13
  4. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  5. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  6. 2024A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Modelscited by 3
  7. 2024Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokenscited by 1
  8. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'acited by 1
  9. 2026Attention-guided Evidence Grounding for Spoken Question Answeringcited by 0
  10. 2026SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Modelscited by 0