# Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models

arXiv: https://arxiv.org/abs/2506.10855 · HTML: https://mlsanity.com/paper/2506.10855/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Self-Supervised Speech Representations are More Phonetic than Semantic](https://mlsanity.com/paper/2406.08619/) (2024)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [THCHS-30 : A Free Chinese Speech Corpus](https://mlsanity.com/paper/1512.01882/) (2015)
- [The Low-Dimensional Linear Geometry of Contextualized Word Representations](https://mlsanity.com/paper/2105.07109/) (2021)
- [Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration](https://mlsanity.com/paper/2306.06232/) (2023)
- [Probing phoneme, language and speaker information in unsupervised speech representations](https://mlsanity.com/paper/2203.16193/) (2022)
- [Do self-supervised speech models develop human-like perception biases?](https://mlsanity.com/paper/2205.15819/) (2022)
- [Encoding of lexical tone in self-supervised models of spoken language](https://mlsanity.com/paper/2403.16865/) (2024)
- [Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces](https://mlsanity.com/paper/2305.12464/) (2023)
- [Wave to Syntax: Probing spoken language models for syntax](https://mlsanity.com/paper/2305.18957/) (2023)
- [What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model](https://mlsanity.com/paper/2306.06524/) (2023)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models](https://mlsanity.com/paper/2408.13678/) (2024)
- [Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations](https://mlsanity.com/paper/2406.09200/) (2024)
- [Untangling in Invariant Speech Recognition](https://mlsanity.com/paper/2003.01787/) (2020)
- [A context-aware knowledge transferring strategy for CTC-based ASR](https://mlsanity.com/paper/2210.06244/) (2022)
- [What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis](https://mlsanity.com/paper/2107.00439/) (2021)
- [Vibravox: A Dataset of French Speech Captured with Body-conduction Audio Sensors](https://mlsanity.com/paper/2407.11828/) (2024)
- [SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition](https://mlsanity.com/paper/2309.16937/) (2023)
- [Analyzing the Representational Geometry of Acoustic Word Embeddings](https://mlsanity.com/paper/2301.03012/) (2023)

## Built on by (most cited first, 3 of 3)

- [A framework for analyzing concept representations in neural models](https://mlsanity.com/paper/2605.01381/) (2026)
- [Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features](https://mlsanity.com/paper/2603.03096/) (2026)
- [Why We Need Speech to Evaluate Speech Translation](https://mlsanity.com/paper/2605.28227/) (2026)

## Abstract (arXiv)

Analyses of self-supervised speech models have begun to reveal where and how they represent different types of information. However, almost all analyses have focused on English. Here, we examine how wav2vec2 models trained on four different languages encode both language-matched and non-matched speech. We use probing classifiers and geometric analyses to examine how phones, lexical tones, and speaker information are represented. We show that for all pretraining and test languages, the subspaces encoding phones, tones, and speakers are largely orthogonal, and that layerwise patterns of probing accuracy are similar, with a relatively small advantage for matched-language phone and tone (but not speaker) probes in the later layers. Our findings suggest that the structure of representations learned by wav2vec2 is largely independent of the speech material used during pretraining.