# What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis

arXiv: https://arxiv.org/abs/2401.17632 · HTML: https://mlsanity.com/paper/2401.17632/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [Similarity Analysis of Contextual Word Representation Models](https://mlsanity.com/paper/2005.01172/) (2020)
- [Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning](https://mlsanity.com/paper/2012.07178/) (2020)
- [Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models](https://mlsanity.com/paper/2207.06867/) (2022)
- [Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?](https://mlsanity.com/paper/2306.00452/) (2023)
- [Similarity Analysis of Self-Supervised Speech Representations](https://mlsanity.com/paper/2010.11481/) (2020)
- [Probing Acoustic Representations for Phonetic Properties](https://mlsanity.com/paper/2010.13007/) (2020)
- [Understanding Self-Attention of Self-Supervised Audio Transformers](https://mlsanity.com/paper/2006.03265/) (2020)
- [Pushing the limits of self-supervised speaker verification using regularized distillation framework](https://mlsanity.com/paper/2211.04168/) (2022)
- [VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/1912.02522/) (2019)
- [Self-supervised speaker embeddings](https://mlsanity.com/paper/1904.03486/) (2019)
- [C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verification](https://mlsanity.com/paper/2208.07446/) (2022)
- [Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning](https://mlsanity.com/paper/2207.05506/) (2022)
- [VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2302.10248/) (2023)
- [SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?](https://mlsanity.com/paper/2306.08374/) (2023)
- [A comprehensive study on self-supervised distillation for speaker representation learning](https://mlsanity.com/paper/2210.15936/) (2022)
- [Insights on Neural Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2205.09456/) (2022)
- [Analysing the Masked predictive coding training criterion for pre-training a Speech Representation Model](https://mlsanity.com/paper/2303.06982/) (2023)

## Built on by (most cited first, 11 of 11)

- [SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection](https://mlsanity.com/paper/2407.18517/) (2024)
- [Property Neurons in Self-Supervised Speech Transformers](https://mlsanity.com/paper/2409.05910/) (2024)
- [Disentangling segmental and prosodic factors to non-native speech comprehensibility](https://mlsanity.com/paper/2408.10997/) (2024)
- [WavRx: a Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Model](https://mlsanity.com/paper/2406.18731/) (2024)
- [Layer-aware TDNN: Speaker Recognition Using Multi-Layer Features from Pre-Trained Models](https://mlsanity.com/paper/2409.07770/) (2024)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech](https://mlsanity.com/paper/2510.08593/) (2025)
- [A framework for analyzing concept representations in neural models](https://mlsanity.com/paper/2605.01381/) (2026)
- [DELULU: Discriminative Embedding Learning Using Latent Units for Speaker-Aware Self-Trained Speech Foundational Model](https://mlsanity.com/paper/2510.17662/) (2025)
- [Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPA](https://mlsanity.com/paper/2608.19084/) (2026)
- [Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters](https://mlsanity.com/paper/2610.05575/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has attracted increased attention for learning meaningful speech representations. Speech SSL models, such as WavLM, employ masked prediction training to encode general-purpose representations. In contrast, speaker SSL models, exemplified by DINO-based models, adopt utterance-level training objectives primarily for speaker representation. Understanding how these models represent information is essential for refining model efficiency and effectiveness. Unlike the various analyses of speech SSL, there has been limited investigation into what information speaker SSL captures and how its representation differs from speech SSL or other fully-supervised speaker models. This paper addresses these fundamental questions. We explore the capacity to capture various speech properties by applying SUPERB evaluation probing tasks to speech and speaker SSL models. We also examine which layers are predominantly utilized for each task to identify differences in how speech is represented. Furthermore, we conduct direct comparisons to measure the similarities between layers within and across models. Our analysis unveils that 1) the capacity to represent content information is somewhat unrelated to enhanced speaker representation, 2) specific layers of speech SSL models would be partly specialized in capturing linguistic information, and 3) speaker SSL models tend to disregard linguistic information but exhibit more sophisticated speaker representation.