mlsanityData to 7 Oct 2026

Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

arXiv 2305.12464 ↗First posted 21 May 2023cs.CL cs.SDcited by 19

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech representations are known to encode both speaker and phonetic information, but how they are distributed in the high-dimensional space remains largely unexplored. We hypothesize that they are encoded in orthogonal subspaces, a property that lends itself to simple disentanglement. Applying principal component analysis to representations of two predictive coding models, we identify two subspaces that capture speaker and phonetic variances, and confirm that they are nearly orthogonal. Based on this property, we propose a new speaker normalization method which collapses the subspace that encodes speaker information, without requiring transcriptions. Probing experiments show that our method effectively eliminates speaker information and outperforms a previous baseline in phone discrimination tasks. Moreover, the approach generalizes and can be used to remove information of unseen speakers.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

12 of 21 references.

References · 12

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  7. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  8. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  9. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 31
  10. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  11. 2022Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Modelscited by 21
  12. 2020Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasetscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 5; 2026: 8.

19 papers.

Sort:

By area:

  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  2. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  3. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  4. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 7
  5. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  6. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  7. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  8. 2025Can Diffusion Models Disentangle? A Theoretical Perspectivecited by 3
  9. 2025LinearVC: Linear transformations of self-supervised features through the lens of voice conversioncited by 3
  10. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  11. 2024A predictive learning model can simulate temporal dynamics and context effects found in neural representations of continuous speechcited by 2
  12. 2023Towards Matching Phones and Speech Representationscited by 1
7 more
  1. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  2. 2026Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition modelscited by 1
  3. 2026Speaker Group Encoding in Self-supervised Speech Recognition Modelscited by 1
  4. 2026Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Featurescited by 0
  5. 2026A framework for analyzing concept representations in neural modelscited by 0
  6. 2026Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPAcited by 0
  7. 2026Do speech foundation models really learn words?cited by 0