# Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model

arXiv: https://arxiv.org/abs/2405.09768 · HTML: https://mlsanity.com/paper/2405.09768/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2024.

## Built on by (most cited first, 8 of 8)

- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge](https://mlsanity.com/paper/2505.23009/) (2025)
- [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024)
- [SF-Speech: Straightened Flow for Zero-Shot Voice Clone](https://mlsanity.com/paper/2410.12399/) (2024)
- [Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis](https://mlsanity.com/paper/2507.02176/) (2025)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)
- [Who Gets the Mic? Investigating Gender Bias in the Speaker Assignment of a Speech-LLM](https://mlsanity.com/paper/2508.13603/) (2025)
- [Acoustic and perceptual differences between standard and accented speech and their voice clones](https://mlsanity.com/paper/2604.01562/) (2026)

## Abstract (arXiv)

Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. These speech language models (SLMs), similarly to their textual counterparts, are scalable, probabilistic, and context-aware. While they can produce diverse and natural outputs, they sometimes face issues such as unintelligibility and the inclusion of non-speech noises or hallucination. As the adoption of this innovative paradigm in speech synthesis increases, there is a clear need for an in-depth evaluation of its capabilities and limitations. In this paper, we evaluate TTS from a discrete token-based SLM, through both automatic metrics and listening tests. We examine five key dimensions: speaking style, intelligibility, speaker consistency, prosodic variation, spontaneous behaviour. Our results highlight the model's strength in generating varied prosody and spontaneous outputs. It is also rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. However, the model's performance in intelligibility and speaker consistency lags behind traditional TTS. Additionally, we show that increasing the scale of SLMs offers a modest boost in robustness. Our findings aim to serve as a benchmark for future advancements in generative SLMs for speech synthesis.