# Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs

arXiv: https://arxiv.org/abs/2508.17863 · HTML: https://mlsanity.com/paper/2508.17863/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities](https://mlsanity.com/paper/2410.11190/) (2024)
- [BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing](https://mlsanity.com/paper/2309.00916/) (2023)
- [A Survey on Speech Large Language Models for Understanding](https://mlsanity.com/paper/2410.18908/) (2024)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [Are discrete units necessary for Spoken Language Modeling?](https://mlsanity.com/paper/2203.05936/) (2022)
- [How Should We Extract Discrete Audio Tokens from Self-Supervised Models?](https://mlsanity.com/paper/2406.10735/) (2024)
- [Comparing Discrete and Continuous Space LLMs for Speech Recognition](https://mlsanity.com/paper/2409.00800/) (2024)
- [ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2504.10344/) (2025)
- [InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training](https://mlsanity.com/paper/2503.02769/) (2025)
- [Exploring the Benefits of Tokenization of Discrete Acoustic Units](https://mlsanity.com/paper/2406.05547/) (2024)
- [Continuous Speech Tokenizer in Text To Speech](https://mlsanity.com/paper/2410.17081/) (2024)

## Built on by (most cited first, 4 of 4)

- [On the Emotion Understanding of Synthesized Speech](https://mlsanity.com/paper/2603.16483/) (2026)
- [Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation](https://mlsanity.com/paper/2606.10368/) (2026)
- [HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models](https://mlsanity.com/paper/2606.27627/) (2026)
- [Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs](https://mlsanity.com/paper/2609.22851/) (2026)

## Abstract (arXiv)

With the rise of Speech Large Language Models (SpeechLLMs), two dominant approaches have emerged for speech processing: discrete tokens and continuous features. Each approach has demonstrated strong capabilities in audio-related processing tasks. However, the performance gap between these two paradigms has not been thoroughly explored. To address this gap, we present a fair comparison of self-supervised learning (SSL)-based discrete and continuous features under the same experimental settings. We evaluate their performance across six spoken language understanding-related tasks using both small and large-scale LLMs (Qwen1.5-0.5B and Llama3.1-8B). We further conduct in-depth analyses, including efficient comparison, SSL layer analysis, LLM layer analysis, and robustness comparison. Our findings reveal that continuous features generally outperform discrete tokens in various tasks. Each speech processing method exhibits distinct characteristics and patterns in how it learns and processes speech information. We hope our results will provide valuable insights to advance spoken language understanding in SpeechLLMs.