# Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equation

arXiv: https://arxiv.org/abs/2505.19273 · HTML: https://mlsanity.com/paper/2505.19273/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 24 of 24)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization](https://mlsanity.com/paper/2012.04456/) (2020)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data](https://mlsanity.com/paper/2402.08093/) (2024)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [Towards Understanding Linear Word Analogies](https://mlsanity.com/paper/1810.04882/) (2018)
- [Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing](https://mlsanity.com/paper/2108.00917/) (2021)
- [ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations](https://mlsanity.com/paper/2302.08137/) (2023)
- [Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations](https://mlsanity.com/paper/2406.09200/) (2024)
- [Vec-Tok Speech: speech vectorization and tokenization for neural speech generation](https://mlsanity.com/paper/2310.07246/) (2023)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations](https://mlsanity.com/paper/2402.03407/) (2024)
- [Voice Cloning: a Multi-Speaker Text-to-Speech Synthesis Approach based on Transfer Learning](https://mlsanity.com/paper/2102.05630/) (2021)
- [Enhancing Polyglot Voices by Leveraging Cross-Lingual Fine-Tuning in Any-to-One Voice Conversion](https://mlsanity.com/paper/2409.17387/) (2024)

## Built on by (most cited first, 3 of 3)

- [LinearVC: Linear transformations of self-supervised features through the lens of voice conversion](https://mlsanity.com/paper/2506.01510/) (2025)
- [SNAP: Speaker Nulling for Artifact Projection in Speech Deepfake Detection](https://mlsanity.com/paper/2603.20686/) (2026)
- [Reusing Latent Speech Representations for Query-Conditioned Topic Localization in Transcripts](https://mlsanity.com/paper/2609.21844/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has reduced the reliance on expensive labeling in speech technologies by learning meaningful representations from unannotated data. Since most SSL-based downstream tasks prioritize content information in speech, ideal representations should disentangle content from unwanted variations like speaker characteristics in the SSL representations. However, removing speaker information often degrades other speech components, and existing methods either fail to fully disentangle speaker identity or require resource-intensive models. In this paper, we propose a novel disentanglement method that linearly decomposes SSL representations into speaker-specific and speaker-independent components, effectively generating speaker disentangled representations. Comprehensive experiments show that our approach achieves speaker independence and as such, when applied to content-driven tasks such as voice conversion, our representations yield significant improvements over state-of-the-art methods.