# Differentially Private Speaker Anonymization

arXiv: https://arxiv.org/abs/2202.11823 · HTML: https://mlsanity.com/paper/2202.11823/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 85 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Certified Robustness to Adversarial Examples with Differential Privacy](https://mlsanity.com/paper/1802.03471/) (2018)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [The Composition Theorem for Differential Privacy](https://mlsanity.com/paper/1311.0776/) (2013)
- [Auditing Differentially Private Machine Learning: How Private is Private SGD?](https://mlsanity.com/paper/2006.07709/) (2020)
- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity](https://mlsanity.com/paper/1811.12469/) (2018)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling](https://mlsanity.com/paper/2012.12803/) (2020)
- [The Privacy Blanket of the Shuffle Model](https://mlsanity.com/paper/1903.02837/) (2019)
- [Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness](https://mlsanity.com/paper/2010.01285/) (2020)
- [Introducing the VoicePrivacy Initiative](https://mlsanity.com/paper/2005.01387/) (2020)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Speaker Anonymization Using X-vector and Neural Waveform Models](https://mlsanity.com/paper/1905.13561/) (2019)
- [Learning Robust and Multilingual Speech Representations](https://mlsanity.com/paper/2001.11128/) (2020)
- [Speaker anonymisation using the McAdams coefficient](https://mlsanity.com/paper/2011.01130/) (2020)
- [Evaluating Voice Conversion-based Privacy Protection against Informed Attackers](https://mlsanity.com/paper/1911.03934/) (2019)
- [Design Choices for X-vector Based Speaker Anonymization](https://mlsanity.com/paper/2005.08601/) (2020)
- [Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?](https://mlsanity.com/paper/1911.04913/) (2019)
- [The State of Speech in HCI: Trends, Themes and Challenges](https://mlsanity.com/paper/1810.06828/) (2018)
- [Training Neural Speech Recognition Systems with Synthetic Speech Augmentation](https://mlsanity.com/paper/1811.00707/) (2018)
- [The GDPR & Speech Data: Reflections of Legal and Technology Communities, First Steps towards a Common Understanding](https://mlsanity.com/paper/1907.03458/) (2019)
- [Preech: A System for Privacy-Preserving Speech Transcription](https://mlsanity.com/paper/1909.04198/) (2019)
- [Voice-Indistinguishability: Protecting Voiceprint in Privacy-Preserving Speech Data Release](https://mlsanity.com/paper/2004.07442/) (2020)
- [A Study of F0 Modification for X-Vector Based Speech Pseudonymization Across Gender](https://mlsanity.com/paper/2101.08478/) (2021)

## Built on by (most cited first, 16 of 16)

- [Privacy-Preserving Instructions for Aligning Large Language Models](https://mlsanity.com/paper/2402.13659/) (2024)
- [The VoicePrivacy 2022 Challenge Evaluation Plan](https://mlsanity.com/paper/2203.12468/) (2022)
- [Detecting Unintended Memorization in Language-Model-Fused ASR](https://mlsanity.com/paper/2204.09606/) (2022)
- [Are disentangled representations all you need to build speaker anonymization systems?](https://mlsanity.com/paper/2208.10497/) (2022)
- [InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy](https://mlsanity.com/paper/2507.02974/) (2025)
- [SAIC: Integration of Speech Anonymization and Identity Classification](https://mlsanity.com/paper/2312.15190/) (2023)
- [A Benchmark for Multi-speaker Anonymization](https://mlsanity.com/paper/2407.05608/) (2024)
- [Probing the Feasibility of Multilingual Speaker Anonymization](https://mlsanity.com/paper/2407.02937/) (2024)
- [The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers](https://mlsanity.com/paper/2401.01537/) (2024)
- [Activity Recognition on Avatar-Anonymized Datasets with Masked Differential Privacy](https://mlsanity.com/paper/2410.17098/) (2024)
- [Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2502.00451/) (2025)
- [Vocoder drift compensation by x-vector alignment in speaker anonymisation](https://mlsanity.com/paper/2307.08403/) (2023)
- [Universal Semantic Disentangled Privacy-preserving Speech Representation Learning](https://mlsanity.com/paper/2505.13085/) (2025)
- [SEF-MK: Speaker-Embedding-Free Voice Anonymization through Multi-k-means Quantization](https://mlsanity.com/paper/2508.07086/) (2025)
- [SegReConcat: A Data Augmentation Method for Voice Anonymization Attack](https://mlsanity.com/paper/2508.18907/) (2025)
- [VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks](https://mlsanity.com/paper/2509.18413/) (2025)

## Abstract (arXiv)

Sharing real-world speech utterances is key to the training and deployment of voice-based services. However, it also raises privacy risks as speech contains a wealth of personal data. Speaker anonymization aims to remove speaker information from a speech utterance while leaving its linguistic and prosodic attributes intact. State-of-the-art techniques operate by disentangling the speaker information (represented via a speaker embedding) from these attributes and re-synthesizing speech based on the speaker embedding of another speaker. Prior research in the privacy community has shown that anonymization often provides brittle privacy protection, even less so any provable guarantee. In this work, we show that disentanglement is indeed not perfect: linguistic and prosodic attributes still contain speaker information. We remove speaker information from these attributes by introducing differentially private feature extractors based on an autoencoder and an automatic speech recognizer, respectively, trained using noise layers. We plug these extractors in the state-of-the-art anonymization pipeline and generate, for the first time, private speech utterances with a provable upper bound on the speaker information they contain. We evaluate empirically the privacy and utility resulting from our differentially private speaker anonymization approach on the LibriSpeech data set. Experimental results show that the generated utterances retain very high utility for automatic speech recognition training and inference, while being much better protected against strong adversaries who leverage the full knowledge of the anonymization process to try to infer the speaker identity.