# On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models

arXiv: https://arxiv.org/abs/2606.23285 · HTML: https://mlsanity.com/paper/2606.23285/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 34 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Textually Pretrained Speech Language Models](https://mlsanity.com/paper/2305.13009/) (2023)
- [LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis](https://mlsanity.com/paper/2505.02625/) (2025)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation](https://mlsanity.com/paper/2401.13527/) (2024)
- [Scaling Properties of Speech Language Models](https://mlsanity.com/paper/2404.00685/) (2024)
- [BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models](https://mlsanity.com/paper/2306.01506/) (2023)
- [SyllableLM: Learning Coarse Semantic Units for Speech Language Models](https://mlsanity.com/paper/2410.04029/) (2024)
- [Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2411.01834/) (2024)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [Sylber: Syllabic Embedding Representation of Speech from Raw Audio](https://mlsanity.com/paper/2410.07168/) (2024)
- [Long-Form Speech Generation with Spoken Language Models](https://mlsanity.com/paper/2412.18603/) (2024)
- [Spoken Language Modeling with Duration-Penalized Self-Supervised Units](https://mlsanity.com/paper/2505.23494/) (2025)
- [Exploring the Benefits of Tokenization of Discrete Acoustic Units](https://mlsanity.com/paper/2406.05547/) (2024)
- [Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information](https://mlsanity.com/paper/2505.15667/) (2025)
- [Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models](https://mlsanity.com/paper/2505.17446/) (2025)

## Abstract (arXiv)

Generative Spoken Language Modeling (GSLM) enables text-free speech modeling by training language models (LMs) using discrete speech representations instead of textual transcription. In this paper, we investigate the performance of GSLM on speech synthesis and continuation using discrete speech representations with varying bitrates. We segment speech representations with fixed widths and train K-means models in multiple cluster sizes, resulting in various bitrate settings. We demonstrate that intelligible and natural speech can be synthesized at lower bitrate settings than the baseline. Furthermore, speech continuation quality remains stable at lower bitrates across multiple metrics, suggesting that the conventional GSLM setting may be redundant for effective speech generation. Although LLM-based metrics show higher correlation with human subjective score than conventional metrics, it remains low, highlighting the need for more stable automatic evaluation methods.