mlsanityData to 14 Oct 2026

Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers

arXiv 2506.21712 ↗First posted 26 Jun 2025cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

In recent years, the impact of self-supervised speech Transformers has extended to speaker-related applications. However, little research has explored how these models encode speaker information. In this work, we address this gap by identifying neurons in the feed-forward layers that are correlated with speaker information. Specifically, we analyze neurons associated with k-means clusters of self-supervised features and i-vectors. Our analysis reveals that these clusters correspond to broad phonetic and gender classes, making them suitable for identifying neurons that represent speakers. By protecting these neurons during pruning, we can significantly preserve performance on speaker-related task, demonstrating their crucial role in encoding speaker information.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

25 of 34 references.

References · 25

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  12. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
13 more
  1. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  2. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  3. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  4. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  5. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  6. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 36
  7. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  8. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  9. 2019Information-Theoretic Understanding of Population Risk Improvement with Model Compressioncited by 11
  10. 2018Speaker Embedding Extraction with Phonetic Informationcited by 10
  11. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
  12. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  13. 2024Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 6