# Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring

arXiv: https://arxiv.org/abs/2202.11929 · HTML: https://mlsanity.com/paper/2202.11929/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 68 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units](https://mlsanity.com/paper/2010.05967/) (2020)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [A segmental framework for fully-unsupervised large-vocabulary speech recognition](https://mlsanity.com/paper/1606.06950/) (2016)
- [Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing](https://mlsanity.com/paper/2108.00917/) (2021)
- [Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks](https://mlsanity.com/paper/2012.07551/) (2020)
- [Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and words](https://mlsanity.com/paper/2110.15909/) (2021)
- [Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection](https://mlsanity.com/paper/1808.02228/) (2018)
- [Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics](https://mlsanity.com/paper/2008.00731/) (2020)
- [Aligned Contrastive Predictive Coding](https://mlsanity.com/paper/2104.11946/) (2021)
- [Self-Expressing Autoencoders for Unsupervised Spoken Term Discovery](https://mlsanity.com/paper/2007.13033/) (2020)
- [Blind phoneme segmentation with temporal prediction errors](https://mlsanity.com/paper/1608.00508/) (2016)
- [Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries](https://mlsanity.com/paper/1703.07588/) (2017)
- [Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery](https://mlsanity.com/paper/1904.03876/) (2019)
- [Unsupervised word segmentation and lexicon discovery using acoustic word embeddings](https://mlsanity.com/paper/1603.02845/) (2016)
- [From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning](https://mlsanity.com/paper/1610.03342/) (2016)
- [Double Articulation Analyzer with Prosody for Unsupervised Word and Phoneme Discovery](https://mlsanity.com/paper/2103.08199/) (2021)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)

## Built on by (most cited first, 24 of 24)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon](https://mlsanity.com/paper/2206.11332/) (2022)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition](https://mlsanity.com/paper/2306.01108/) (2023)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Spoken Language Modeling with Duration-Penalized Self-Supervised Units](https://mlsanity.com/paper/2505.23494/) (2025)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Revisiting speech segmentation and lexicon learning with better features](https://mlsanity.com/paper/2401.17902/) (2024)
- [Investigating the 'Autoencoder Behavior' in Speech Self-Supervised Models: a focus on HuBERT's Pretraining](https://mlsanity.com/paper/2405.08402/) (2024)
- [Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences](https://mlsanity.com/paper/2303.08809/) (2023)
- [Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings](https://mlsanity.com/paper/2307.02083/) (2023)
- [Audio-Visual Neural Syntax Acquisition](https://mlsanity.com/paper/2310.07654/) (2023)
- [A Temporal Extension of Latent Dirichlet Allocation for Unsupervised Acoustic Unit Discovery](https://mlsanity.com/paper/2206.11706/) (2022)
- [XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words](https://mlsanity.com/paper/2310.05235/) (2023)
- [Towards Unsupervised Speech Recognition Without Pronunciation Models](https://mlsanity.com/paper/2406.08380/) (2024)
- [A Simple HMM with Self-Supervised Representations for Phone Segmentation](https://mlsanity.com/paper/2409.09646/) (2024)
- [Unsupervised Speech Segmentation: A General Approach Using Speech Language Models](https://mlsanity.com/paper/2501.03711/) (2025)
- [DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units](https://mlsanity.com/paper/2603.18612/) (2026)
- [Recovering the Zipfian Distribution in Unsupervised Term Discovery](https://mlsanity.com/paper/2606.10781/) (2026)

## Abstract (arXiv)

Recent work on unsupervised speech segmentation has used self-supervised models with phone and word segmentation modules that are trained jointly. This paper instead revisits an older approach to word segmentation: bottom-up phone-like unit discovery is performed first, and symbolic word segmentation is then performed on top of the discovered units (without influencing the lower level). To do this, I propose a new unit discovery model, a new symbolic word segmentation model, and then chain the two models to segment speech. Both models use dynamic programming to minimize segment costs from a self-supervised network with an additional duration penalty that encourages longer units. Concretely, for acoustic unit discovery, duration-penalized dynamic programming (DPDP) is used with a contrastive predictive coding model as the scoring network. For word segmentation, DPDP is applied with an autoencoding recurrent neural as the scoring network. The two models are chained in order to segment speech. This approach gives comparable word segmentation results to state-of-the-art joint self-supervised segmentation models on an English benchmark. On French, Mandarin, German and Wolof data, it outperforms previous systems on the ZeroSpeech benchmarks. Analysis shows that the chained DPDP system segments shorter filler words well, but longer words might require some external top-down signal.