# Unsupervised Speech Segmentation: A General Approach Using Speech Language Models

arXiv: https://arxiv.org/abs/2501.03711 · HTML: https://mlsanity.com/paper/2501.03711/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 18 of 18)

- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Textually Pretrained Speech Language Models](https://mlsanity.com/paper/2305.13009/) (2023)
- [The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems](https://mlsanity.com/paper/1806.09514/) (2018)
- [Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling](https://mlsanity.com/paper/2301.00591/) (2023)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [textless-lib: a Library for Textless Spoken Language Processing](https://mlsanity.com/paper/2202.07359/) (2022)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [Speech Emotion Diarization: Which Emotion Appears When?](https://mlsanity.com/paper/2306.12991/) (2023)
- [Phoneme Boundary Detection using Learnable Segmental Features](https://mlsanity.com/paper/2002.04992/) (2020)
- [Topical Segmentation of Spoken Narratives: A Test Case on Holocaust Survivor Testimonies](https://mlsanity.com/paper/2210.13783/) (2022)
- [Automatic measurement of vowel duration via structured prediction](https://mlsanity.com/paper/1610.08166/) (2016)
- [Sequence Segmentation Using Joint RNN and Structured Prediction Models](https://mlsanity.com/paper/1610.07918/) (2016)

## Built on by (most cited first, 1 of 1)

- [Slamming: Training a Speech Language Model on One GPU in a Day](https://mlsanity.com/paper/2502.15814/) (2025)

## Abstract (arXiv)

In this paper, we introduce an unsupervised approach for Speech Segmentation, which builds on previously researched approaches, e.g., Speaker Diarization, while being applicable to an inclusive set of acoustic-semantic distinctions, paving a path towards a general Unsupervised Speech Segmentation approach. Unlike traditional speech and audio segmentation, which mainly focuses on spectral changes in the input signal, e.g., phone segmentation, our approach tries to segment the spoken utterance into chunks with differing acoustic-semantic styles, focusing on acoustic-semantic information that does not translate well into text, e.g., emotion or speaker. While most Speech Segmentation tasks only handle one style change, e.g., emotion diarization, our approach tries to handle multiple acoustic-semantic style changes. Leveraging recent advances in Speech Language Models (SLMs), we propose a simple unsupervised method to segment a given speech utterance. We empirically demonstrate the effectiveness of the proposed approach by considering several setups. Results suggest that the proposed method is superior to the evaluated baselines on boundary detection, segment purity, and over-segmentation. Code is available at https://github.com/avishaiElmakies/unsupervised_speech_segmentation_using_slm.