# An embedded segmental K-means model for unsupervised segmentation and clustering of speech

arXiv: https://arxiv.org/abs/1703.08135 · HTML: https://mlsanity.com/paper/1703.08135/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2017.

## Built on by (most cited first, 30 of 31)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [End-to-End Simultaneous Speech Translation with Differentiable Segmentation](https://mlsanity.com/paper/2305.16093/) (2023)
- [Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models](https://mlsanity.com/paper/2210.16043/) (2022)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT](https://mlsanity.com/paper/2310.10803/) (2023)
- [DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon](https://mlsanity.com/paper/2206.11332/) (2022)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labels](https://mlsanity.com/paper/2304.00993/) (2023)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Unified Segment-to-Segment Framework for Simultaneous Sequence Generation](https://mlsanity.com/paper/2310.17940/) (2023)
- [Towards Audio Token Compression in Large Audio Language Models](https://mlsanity.com/paper/2511.20973/) (2025)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds](https://mlsanity.com/paper/2402.03269/) (2024)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Towards Tokenized Human Dynamics Representation](https://mlsanity.com/paper/2111.11433/) (2021)
- [Revisiting speech segmentation and lexicon learning with better features](https://mlsanity.com/paper/2401.17902/) (2024)
- [U-vectors: Generating clusterable speaker embedding from unlabeled data](https://mlsanity.com/paper/2102.03868/) (2021)
- [Unsupervised Acoustic Unit Discovery by Leveraging a Language-Independent Subword Discriminative Feature Representation](https://mlsanity.com/paper/2104.00994/) (2021)
- [Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings](https://mlsanity.com/paper/2106.04298/) (2021)
- [Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings](https://mlsanity.com/paper/2307.02083/) (2023)
- [A Simple HMM with Self-Supervised Representations for Phone Segmentation](https://mlsanity.com/paper/2409.09646/) (2024)
- [Revisiting Lexicon Evaluation in Unsupervised Word Discovery](https://mlsanity.com/paper/2606.06183/) (2026)

## Abstract (arXiv)

Unsupervised segmentation and clustering of unlabelled speech are core problems in zero-resource speech processing. Most approaches lie at methodological extremes: some use probabilistic Bayesian models with convergence guarantees, while others opt for more efficient heuristic techniques. Despite competitive performance in previous work, the full Bayesian approach is difficult to scale to large speech corpora. We introduce an approximation to a recent Bayesian model that still has a clear objective function but improves efficiency by using hard clustering and segmentation rather than full Bayesian inference. Like its Bayesian counterpart, this embedded segmental K-means model (ES-KMeans) represents arbitrary-length word segments as fixed-dimensional acoustic word embeddings. We first compare ES-KMeans to previous approaches on common English and Xitsonga data sets (5 and 2.5 hours of speech): ES-KMeans outperforms a leading heuristic method in word segmentation, giving similar scores to the Bayesian model while being 5 times faster with fewer hyperparameters. However, its clusters are less pure than those of the other models. We then show that ES-KMeans scales to larger corpora by applying it to the 5 languages of the Zero Resource Speech Challenge 2017 (up to 45 hours), where it performs competitively compared to the challenge baseline.