# Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

arXiv: https://arxiv.org/abs/2305.12464 · HTML: https://mlsanity.com/paper/2305.12464/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 21 May 2023.

## Builds on (most cited first, 12 of 12)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Probing phoneme, language and speaker information in unsupervised speech representations](https://mlsanity.com/paper/2203.16193/) (2022)
- [Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing](https://mlsanity.com/paper/2108.00917/) (2021)
- [Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models](https://mlsanity.com/paper/2210.16043/) (2022)
- [Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets](https://mlsanity.com/paper/2007.04205/) (2020)

## Built on by (most cited first, 20 of 20)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment](https://mlsanity.com/paper/2502.07029/) (2025)
- [Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations](https://mlsanity.com/paper/2406.09200/) (2024)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces](https://mlsanity.com/paper/2603.12642/) (2026)
- [Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget](https://mlsanity.com/paper/2409.16295/) (2024)
- [Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?](https://mlsanity.com/paper/2410.19935/) (2024)
- [Can Diffusion Models Disentangle? A Theoretical Perspective](https://mlsanity.com/paper/2504.00220/) (2025)
- [LinearVC: Linear transformations of self-supervised features through the lens of voice conversion](https://mlsanity.com/paper/2506.01510/) (2025)
- [Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models](https://mlsanity.com/paper/2506.10855/) (2025)
- [A predictive learning model can simulate temporal dynamics and context effects found in neural representations of continuous speech](https://mlsanity.com/paper/2405.08237/) (2024)
- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [A framework for analyzing concept representations in neural models](https://mlsanity.com/paper/2605.01381/) (2026)
- [Towards Matching Phones and Speech Representations](https://mlsanity.com/paper/2310.17558/) (2023)
- [Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features](https://mlsanity.com/paper/2603.03096/) (2026)
- [Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priors](https://mlsanity.com/paper/2603.08359/) (2026)
- [Speaker Group Encoding in Self-supervised Speech Recognition Models](https://mlsanity.com/paper/2606.10654/) (2026)
- [Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPA](https://mlsanity.com/paper/2608.19084/) (2026)
- [Do speech foundation models really learn words?](https://mlsanity.com/paper/2609.10434/) (2026)
- [Deep Speech Representations Track Word Timing along a Low-Dimensional Circular Trajectory](https://mlsanity.com/paper/2610.16273/) (2026)

## Abstract (arXiv)

Self-supervised speech representations are known to encode both speaker and phonetic information, but how they are distributed in the high-dimensional space remains largely unexplored. We hypothesize that they are encoded in orthogonal subspaces, a property that lends itself to simple disentanglement. Applying principal component analysis to representations of two predictive coding models, we identify two subspaces that capture speaker and phonetic variances, and confirm that they are nearly orthogonal. Based on this property, we propose a new speaker normalization method which collapses the subspace that encodes speaker information, without requiring transcriptions. Probing experiments show that our method effectively eliminates speaker information and outperforms a previous baseline in phone discrimination tasks. Moreover, the approach generalizes and can be used to remove information of unseen speakers.