mlsanityData to 7 Oct 2026

Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations

arXiv 2406.09200 ↗First posted 13 Jun 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech representations can hugely benefit downstream speech technologies, yet the properties that make them useful are still poorly understood. Two candidate properties related to the geometry of the representation space have been hypothesized to correlate well with downstream tasks: (1) the degree of orthogonality between the subspaces spanned by the speaker centroids and phone centroids, and (2) the isotropy of the space, i.e., the degree to which all dimensions are effectively utilized. To study them, we introduce a new measure, Cumulative Residual Variance (CRV), which can be used to assess both properties. Using linear classifiers for speaker and phone ID to probe the representations of six different self-supervised models and two untrained baselines, we ask whether either orthogonality or isotropy correlate with linear probing accuracy. We find that both measures correlate with phonetic probing accuracy, though our results on isotropy are more nuanced.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

22 of 35 references.

References · 22

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  8. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  9. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  10. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  11. 2022The Geometry of Multilingual Language Model Representationscited by 94
  12. 2020Fairness by Learning Orthogonal Disentangled Representationscited by 76
10 more
  1. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  2. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  3. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  4. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  5. 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 19
  6. 2022Opening the Black Box of wav2vec Feature Encodercited by 18
  7. 2020Untangling in Invariant Speech Recognitioncited by 12
  8. 2023Stable Anisotropic Regularizationcited by 12
  9. 2021Do Acoustic Word Embeddings Capture Phonological Similarity? An Empirical Studycited by 8
  10. 2023Analyzing the Representational Geometry of Acoustic Word Embeddingscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

  1. 2025The Role of Prosody in Spoken Question Answeringcited by 6
  2. 2025Effective Context in Neural Speech Modelscited by 5
  3. 2025Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0cited by 4
  4. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  5. 2025Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equationcited by 3
  6. 2025LinearVC: Linear transformations of self-supervised features through the lens of voice conversioncited by 3
  7. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  8. 2025Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformerscited by 1
  9. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  10. 2026Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition modelscited by 1
  11. 2026A framework for analyzing concept representations in neural modelscited by 0
  12. 2026Learning task-specific subspaces via interventional post-training of speech foundation modelscited by 0
2 more
  1. 2026Probing in the Wild: A Case Study of Self-Supervised Speech Representations on Mandarin Sub-dialects with Unsupervised Articulatory Analysiscited by 0
  2. 2026Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPAcited by 0