# Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and words

arXiv: https://arxiv.org/abs/2110.15909 · HTML: https://mlsanity.com/paper/2110.15909/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks](https://mlsanity.com/paper/2012.07551/) (2020)
- [Aligned Contrastive Predictive Coding](https://mlsanity.com/paper/2104.11946/) (2021)

## Built on by (most cited first, 18 of 18)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon](https://mlsanity.com/paper/2206.11332/) (2022)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labels](https://mlsanity.com/paper/2304.00993/) (2023)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Towards Audio Token Compression in Large Audio Language Models](https://mlsanity.com/paper/2511.20973/) (2025)
- [Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors](https://mlsanity.com/paper/2207.00760/) (2022)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings](https://mlsanity.com/paper/2307.02083/) (2023)
- [Towards Unsupervised Speech Recognition Without Pronunciation Models](https://mlsanity.com/paper/2406.08380/) (2024)
- [A Simple HMM with Self-Supervised Representations for Phone Segmentation](https://mlsanity.com/paper/2409.09646/) (2024)
- [Recovering the Zipfian Distribution in Unsupervised Term Discovery](https://mlsanity.com/paper/2606.10781/) (2026)

## Abstract (arXiv)

We investigate the performance on phoneme categorization and phoneme and word segmentation of several self-supervised learning (SSL) methods based on Contrastive Predictive Coding (CPC). Our experiments show that with the existing algorithms there is a trade off between categorization and segmentation performance. We investigate the source of this conflict and conclude that the use of context building networks, albeit necessary for superior performance on categorization tasks, harms segmentation performance by causing a temporal shift on the learned representations. Aiming to bridge this gap, we take inspiration from the leading approach on segmentation, which simultaneously models the speech signal at the frame and phoneme level, and incorporate multi-level modelling into Aligned CPC (ACPC), a variation of CPC which exhibits the best performance on categorization tasks. Our multi-level ACPC (mACPC) improves in all categorization metrics and achieves state-of-the-art performance in word segmentation.