# Adversarial Disentanglement of Speaker Representation for Attribute-Driven Privacy Preservation

arXiv: https://arxiv.org/abs/2012.04454 · HTML: https://mlsanity.com/paper/2012.04454/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 13 of 13)

- [Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy](https://mlsanity.com/paper/2210.07002/) (2022)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Protecting gender and identity with disentangled speech representations](https://mlsanity.com/paper/2104.11051/) (2021)
- [Generating gender-ambiguous voices for privacy-preserving speech recognition](https://mlsanity.com/paper/2207.01052/) (2022)
- [Prosody-Driven Privacy-Preserving Dementia Detection](https://mlsanity.com/paper/2407.03470/) (2024)
- [Adversarial Representation Learning for Robust Privacy Preservation in Audio](https://mlsanity.com/paper/2305.00011/) (2023)
- [Representation Learning for Audio Privacy Preservation using Source Separation and Robust Adversarial Learning](https://mlsanity.com/paper/2308.04960/) (2023)
- [VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks](https://mlsanity.com/paper/2509.18413/) (2025)
- [Beyond Voice Identity Conversion: Manipulating Voice Attributes by Adversarial Learning of Structured Disentangled Representations](https://mlsanity.com/paper/2107.12346/) (2021)
- [Beyond Neural-on-Neural Approaches to Speaker Gender Protection](https://mlsanity.com/paper/2306.17700/) (2023)
- [Privacy-Preserving End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2403.15510/) (2024)
- [ClaritySpeech: Dementia Obfuscation in Speech](https://mlsanity.com/paper/2507.09282/) (2025)
- [A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models](https://mlsanity.com/paper/2609.18533/) (2026)

## Abstract (arXiv)

In speech technologies, speaker's voice representation is used in many applications such as speech recognition, voice conversion, speech synthesis and, obviously, user authentication. Modern vocal representations of the speaker are based on neural embeddings. In addition to the targeted information, these representations usually contain sensitive information about the speaker, like the age, sex, physical state, education level or ethnicity. In order to allow the user to choose which information to protect, we introduce in this paper the concept of attribute-driven privacy preservation in speaker voice representation. It allows a person to hide one or more personal aspects to a potential malicious interceptor and to the application provider. As a first solution to this concept, we propose to use an adversarial autoencoding method that disentangles in the voice representation a given speaker attribute thus allowing its concealment. We focus here on the sex attribute for an Automatic Speaker Verification (ASV) task. Experiments carried out using the VoxCeleb datasets have shown that the proposed method enables the concealment of this attribute while preserving ASV ability.