# Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation

arXiv: https://arxiv.org/abs/2106.02170 · HTML: https://mlsanity.com/paper/2106.02170/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 14 of 14)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [An embedded segmental K-means model for unsupervised segmentation and clustering of speech](https://mlsanity.com/paper/1703.08135/) (2017)
- [A segmental framework for fully-unsupervised large-vocabulary speech recognition](https://mlsanity.com/paper/1606.06950/) (2016)
- [Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks](https://mlsanity.com/paper/2012.07551/) (2020)
- [Blind phoneme segmentation with temporal prediction errors](https://mlsanity.com/paper/1608.00508/) (2016)
- [Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries](https://mlsanity.com/paper/1703.07588/) (2017)

## Built on by (most cited first, 30 of 32)

- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and words](https://mlsanity.com/paper/2110.15909/) (2021)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon](https://mlsanity.com/paper/2206.11332/) (2022)
- [Evaluating context-invariance in unsupervised speech representations](https://mlsanity.com/paper/2210.15775/) (2022)
- [Phoneme Segmentation Using Self-Supervised Speech Models](https://mlsanity.com/paper/2211.01461/) (2022)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Autoregressive Co-Training for Learning Discrete Speech Representations](https://mlsanity.com/paper/2203.15840/) (2022)
- [On Compressing Sequences for Self-Supervised Speech Models](https://mlsanity.com/paper/2210.07189/) (2022)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labels](https://mlsanity.com/paper/2304.00993/) (2023)
- [R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces](https://mlsanity.com/paper/2311.09117/) (2023)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Towards Audio Token Compression in Large Audio Language Models](https://mlsanity.com/paper/2511.20973/) (2025)
- [Unsupervised Speech Recognition with N-Skipgram and Positional Unigram Matching](https://mlsanity.com/paper/2310.02382/) (2023)
- [REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR](https://mlsanity.com/paper/2402.03988/) (2024)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Unsupervised lexicon learning from speech is limited by representations rather than clustering](https://mlsanity.com/paper/2510.09225/) (2025)
- [Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences](https://mlsanity.com/paper/2303.08809/) (2023)
- [Audio-Visual Neural Syntax Acquisition](https://mlsanity.com/paper/2310.07654/) (2023)
- [Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models](https://mlsanity.com/paper/2210.16659/) (2022)
- [XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words](https://mlsanity.com/paper/2310.05235/) (2023)
- [Removing Speaker Information from Speech Representation using Variable-Length Soft Pooling](https://mlsanity.com/paper/2404.00856/) (2024)
- [Smooth InfoMax -- Towards Easier Post-Hoc Interpretability](https://mlsanity.com/paper/2408.12936/) (2024)

## Abstract (arXiv)

Automatic detection of phoneme or word-like units is one of the core objectives in zero-resource speech processing. Recent attempts employ self-supervised training methods, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. However, CPC only looks at the audio signal's frame-level structure. We overcome this limitation with a segmental contrastive predictive coding (SCPC) framework that can model the signal structure at a higher level e.g. at the phoneme level. In this framework, a convolutional neural network learns frame-level representation from the raw waveform via noise-contrastive estimation (NCE). A differentiable boundary detector finds variable-length segments, which are then used to optimize a segment encoder via NCE to learn segment representations. The differentiable boundary detector allows us to train frame-level and segment-level encoders jointly. Typically, phoneme and word segmentation are treated as separate tasks. We unify them and experimentally show that our single model outperforms existing phoneme and word segmentation methods on TIMIT and Buckeye datasets. We analyze the impact of boundary threshold and when is the right time to include the segmental loss in the learning process.