# Evaluating context-invariance in unsupervised speech representations

arXiv: https://arxiv.org/abs/2210.15775 · HTML: https://mlsanity.com/paper/2210.15775/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [Are discrete units necessary for Spoken Language Modeling?](https://mlsanity.com/paper/2203.05936/) (2022)
- [Evaluating the reliability of acoustic speech embeddings](https://mlsanity.com/paper/2007.13542/) (2020)
- [Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics](https://mlsanity.com/paper/2008.00731/) (2020)
- [Textual Supervision for Visually Grounded Spoken Language Understanding](https://mlsanity.com/paper/2010.02806/) (2020)
- [ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition](https://mlsanity.com/paper/2107.06546/) (2021)
- [Discrete representations in neural models of spoken language](https://mlsanity.com/paper/2105.05582/) (2021)

## Built on by (most cited first, 11 of 11)

- [Sylber: Syllabic Embedding Representation of Speech from Raw Audio](https://mlsanity.com/paper/2410.07168/) (2024)
- [Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach](https://mlsanity.com/paper/2410.00025/) (2024)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [fastabx: A library for efficient computation of ABX discriminability](https://mlsanity.com/paper/2505.02692/) (2025)
- [Spoken Language Modeling with Duration-Penalized Self-Supervised Units](https://mlsanity.com/paper/2505.23494/) (2025)
- [Bigger is not Always Better: The Effect of Context Size on Speech Pre-Training](https://mlsanity.com/paper/2312.01515/) (2023)
- [MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery](https://mlsanity.com/paper/2512.19612/) (2025)
- [SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation](https://mlsanity.com/paper/2512.21204/) (2025)
- [Simulating Articulatory Trajectories with Phonological Feature Interpolation](https://mlsanity.com/paper/2408.04363/) (2024)
- [Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks](https://mlsanity.com/paper/2505.17747/) (2025)
- [FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs](https://mlsanity.com/paper/2509.11425/) (2025)

## Abstract (arXiv)

Unsupervised speech representations have taken off, with benchmarks (SUPERB, ZeroSpeech) demonstrating major progress on semi-supervised speech recognition, speech synthesis, and speech-only language modelling. Inspiration comes from the promise of ``discovering the phonemes'' of a language or a similar low-bitrate encoding. However, one of the critical properties of phoneme transcriptions is context-invariance: the phonetic context of a speech sound can have massive influence on the way it is pronounced, while the text remains stable. This is what allows tokens of the same word to have the same transcriptions -- key to language understanding. Current benchmarks do not measure context-invariance. We develop a new version of the ZeroSpeech ABX benchmark that measures context-invariance, and apply it to recent self-supervised representations. We demonstrate that the context-independence of representations is predictive of the stability of word-level representations. We suggest research concentrate on improving context-independence of self-supervised and unsupervised representations.