# A Benchmark for Multi-speaker Anonymization

arXiv: https://arxiv.org/abs/2407.05608 · HTML: https://mlsanity.com/paper/2407.05608/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 76 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 30 of 41)

- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [Speaker Recognition Based on Deep Learning: An Overview](https://mlsanity.com/paper/2012.00931/) (2020)
- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Speaker Anonymization Using X-vector and Neural Waveform Models](https://mlsanity.com/paper/1905.13561/) (2019)
- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [Speaker anonymisation using the McAdams coefficient](https://mlsanity.com/paper/2011.01130/) (2020)
- [Design Choices for X-vector Based Speaker Anonymization](https://mlsanity.com/paper/2005.08601/) (2020)
- [Defending Your Voice: Adversarial Attack on Voice Conversion](https://mlsanity.com/paper/2005.08781/) (2020)
- [Neural source-filter-based waveform model for statistical parametric speech synthesis](https://mlsanity.com/paper/1810.11946/) (2018)
- [Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques](https://mlsanity.com/paper/2308.04455/) (2023)
- [Speaker anonymization using neural audio codec language models](https://mlsanity.com/paper/2309.14129/) (2023)
- [Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions](https://mlsanity.com/paper/2009.03554/) (2020)
- [The VoicePrivacy 2022 Challenge Evaluation Plan](https://mlsanity.com/paper/2203.12468/) (2022)
- [Speaker anonymization using orthogonal Householder neural network](https://mlsanity.com/paper/2305.18823/) (2023)
- [LSTM based Similarity Measurement with Spectral Clustering for Speaker Diarization](https://mlsanity.com/paper/1907.10393/) (2019)

## Built on by (most cited first, 7 of 7)

- [The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization](https://mlsanity.com/paper/2601.11846/) (2026)
- [Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2502.00451/) (2025)
- [Protecting Bystander Privacy via Selective Hearing in Audio LLMs](https://mlsanity.com/paper/2512.06380/) (2025)
- [SEF-MK: Speaker-Embedding-Free Voice Anonymization through Multi-k-means Quantization](https://mlsanity.com/paper/2508.07086/) (2025)
- [SegReConcat: A Data Augmentation Method for Voice Anonymization Attack](https://mlsanity.com/paper/2508.18907/) (2025)
- [Target speaker anonymization in multi-speaker recordings](https://mlsanity.com/paper/2510.09307/) (2025)
- [Voice Privacy from an Attribute-based Perspective](https://mlsanity.com/paper/2603.20301/) (2026)

## Abstract (arXiv)

Privacy-preserving voice protection approaches primarily suppress privacy-related information derived from paralinguistic attributes while preserving the linguistic content. Existing solutions focus particularly on single-speaker scenarios. However, they lack practicality for real-world applications, i.e., multi-speaker scenarios. In this paper, we present an initial attempt to provide a multi-speaker anonymization benchmark by defining the task and evaluation protocol, proposing benchmarking solutions, and discussing the privacy leakage of overlapping conversations. The proposed benchmark solutions are based on a cascaded system that integrates spectral-clustering-based speaker diarization and disentanglement-based speaker anonymization using a selection-based anonymizer. To improve utility, the benchmark solutions are further enhanced by two conversation-level speaker vector anonymization methods. The first method minimizes the differential similarity across speaker pairs in the original and anonymized conversations, which maintains original speaker relationships in the anonymized version. The other minimizes the aggregated similarity across anonymized speakers, which achieves better differentiation between speakers.Experiments conducted on both non-overlap simulated and real-world datasets demonstrate the effectiveness of the multi-speaker anonymization system with the proposed speaker anonymizers. Additionally, we analyzed overlapping speech regarding privacy leakage and provided potential solutions