mlsanityData to 7 Oct 2026

Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features

arXiv 2603.03096 ↗First posted 3 Mar 2026eess.AS cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

How do speech models trained through self-supervised learning structure their representations? Previous studies have looked at how information is encoded in feature vectors across different layers. But few studies have considered whether speech characteristics are captured within individual dimensions of SSL features. In this paper we specifically look at speaker information using PCA on utterance-averaged representations. For a range of SSL models, we find that the principal dimension that explains most variance encodes pitch and associated characteristics like gender. Other individual principal dimensions correlate with intensity, noise levels, the second formant, and higher frequency characteristics. We then use synthesis analyses to show that the dimensions for most characteristics are isolated from each other's influence. We further show that characteristics can be changed by manipulating the corresponding dimensions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

18 of 34 references.

References · 18

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  8. 2023Voice Conversion With Just Nearest Neighborscited by 67
  9. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  10. 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 19
  11. 2025A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representationscited by 5
  12. 2025Children's Voice Privacy: First Steps And Emerging Challengescited by 5
6 more
  1. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  2. 2025Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?cited by 4
  3. 2025Unsupervised lexicon learning from speech is limited by representations rather than clusteringcited by 4
  4. 2025LinearVC: Linear transformations of self-supervised features through the lens of voice conversioncited by 3
  5. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3
  6. 2025Analyzing and Improving Speaker Similarity Assessment for Speech Synthesiscited by 3

Built on by

Not cited yet.