# Do speech foundation models really learn words?

arXiv: https://arxiv.org/abs/2609.10434 · HTML: https://mlsanity.com/paper/2609.10434/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [Self-Supervised Speech Representations are More Phonetic than Semantic](https://mlsanity.com/paper/2406.08619/) (2024)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration](https://mlsanity.com/paper/2306.06232/) (2023)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing](https://mlsanity.com/paper/2108.00917/) (2021)
- [Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces](https://mlsanity.com/paper/2305.12464/) (2023)
- [Speech language models lack important brain-relevant semantics](https://mlsanity.com/paper/2311.04664/) (2023)
- [Wave to Syntax: Probing spoken language models for syntax](https://mlsanity.com/paper/2305.18957/) (2023)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?](https://mlsanity.com/paper/2507.19204/) (2025)
- [Discrete Speech Unit Extraction via Independent Component Analysis](https://mlsanity.com/paper/2501.06562/) (2025)
- [Measuring Orthogonality in Representations of Generative Models](https://mlsanity.com/paper/2407.03728/) (2024)

## Abstract (arXiv)

Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.