# Protecting gender and identity with disentangled speech representations

arXiv: https://arxiv.org/abs/2104.11051 · HTML: https://mlsanity.com/paper/2104.11051/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Builds on (most cited first, 21 of 21)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [On the Fairness of Disentangled Representations](https://mlsanity.com/paper/1905.13662/) (2019)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data](https://mlsanity.com/paper/1709.07902/) (2017)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [Introducing the VoicePrivacy Initiative](https://mlsanity.com/paper/2005.01387/) (2020)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [Evaluating Voice Conversion-based Privacy Protection against Informed Attackers](https://mlsanity.com/paper/1911.03934/) (2019)
- [Design Choices for X-vector Based Speaker Anonymization](https://mlsanity.com/paper/2005.08601/) (2020)
- [Privacy-preserving Voice Analysis via Disentangled Representations](https://mlsanity.com/paper/2007.15064/) (2020)
- [Defending Your Voice: Adversarial Attack on Voice Conversion](https://mlsanity.com/paper/2005.08781/) (2020)
- [Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?](https://mlsanity.com/paper/1911.04913/) (2019)
- [Adversarial Disentanglement of Speaker Representation for Attribute-Driven Privacy Preservation](https://mlsanity.com/paper/2012.04454/) (2020)
- [Adversarial representation learning for private speech generation](https://mlsanity.com/paper/2006.09114/) (2020)
- [On Using Backpropagation for Speech Texture Generation and Voice Conversion](https://mlsanity.com/paper/1712.08363/) (2017)

## Built on by (most cited first, 5 of 5)

- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Generating gender-ambiguous voices for privacy-preserving speech recognition](https://mlsanity.com/paper/2207.01052/) (2022)
- [Safeguarding Privacy in Edge Speech Understanding with Tiny Foundation Models](https://mlsanity.com/paper/2502.01649/) (2025)
- [Beyond Neural-on-Neural Approaches to Speaker Gender Protection](https://mlsanity.com/paper/2306.17700/) (2023)
- [Privacy-Preserving End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2403.15510/) (2024)

## Abstract (arXiv)

Besides its linguistic content, our speech is rich in biometric information that can be inferred by classifiers. Learning privacy-preserving representations for speech signals enables downstream tasks without sharing unnecessary, private information about an individual. In this paper, we show that protecting gender information in speech is more effective than modelling speaker-identity information only when generating a non-sensitive representation of speech. Our method relies on reconstructing speech by decoding linguistic content along with gender information using a variational autoencoder. Specifically, we exploit disentangled representation learning to encode information about different attributes into separate subspaces that can be factorised independently. We present a novel way to encode gender information and disentangle two sensitive biometric identifiers, namely gender and identity, in a privacy-protecting setting. Experiments on the LibriSpeech dataset show that gender recognition and speaker verification can be reduced to a random guess, protecting against classification-based attacks.