# Revisiting Lexicon Evaluation in Unsupervised Word Discovery

arXiv: https://arxiv.org/abs/2606.06183 · HTML: https://mlsanity.com/paper/2606.06183/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units](https://mlsanity.com/paper/2010.05967/) (2020)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [A segmental framework for fully-unsupervised large-vocabulary speech recognition](https://mlsanity.com/paper/1606.06950/) (2016)
- [SyllableLM: Learning Coarse Semantic Units for Speech Language Models](https://mlsanity.com/paper/2410.04029/) (2024)
- [Sylber: Syllabic Embedding Representation of Speech from Raw Audio](https://mlsanity.com/paper/2410.07168/) (2024)
- [Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics](https://mlsanity.com/paper/2008.00731/) (2020)
- [Self-Expressing Autoencoders for Unsupervised Spoken Term Discovery](https://mlsanity.com/paper/2007.13033/) (2020)
- [Spoken-Term Discovery using Discrete Speech Units](https://mlsanity.com/paper/2408.14390/) (2024)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Revisiting speech segmentation and lexicon learning with better features](https://mlsanity.com/paper/2401.17902/) (2024)

## Built on by (most cited first, 1 of 1)

- [Recovering the Zipfian Distribution in Unsupervised Term Discovery](https://mlsanity.com/paper/2606.10781/) (2026)

## Abstract (arXiv)

Building a lexicon from discovered word-like units is a central goal in zero-resource speech processing. But do our evaluations provide a trustworthy indication of lexicon quality? A common metric, normalized edit distance, averages the phoneme edit distances between discovered units in each cluster. We show that this metric has an inherent bias toward the quality of large clusters, inhibiting fair evaluation. Moreover, it ignores how well true classes are distributed across clusters. Based on established theory in clustering literature, we propose two metrics that address these shortcomings: a modified metric that weighs cluster size when assessing within-cluster consistency, and an inverse metric that assesses how true words are spread across clusters. Through experiments on synthetic and real-world lexicons, we demonstrate that combined, these metrics are: (1) more closely correlated with how similar a lexicon is to the ground-truth distribution, and (2) more robust to biases that skew lexicon evaluations.