# VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks

arXiv: https://arxiv.org/abs/2509.18413 · HTML: https://mlsanity.com/paper/2509.18413/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 15 of 15)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques](https://mlsanity.com/paper/2308.04455/) (2023)
- [Scaling Rich Style-Prompted Text-to-Speech Datasets](https://mlsanity.com/paper/2503.04713/) (2025)
- [Differentially Private Speaker Anonymization](https://mlsanity.com/paper/2202.11823/) (2022)
- [Bounding Membership Inference](https://mlsanity.com/paper/2202.12232/) (2022)
- [Adversarial Disentanglement of Speaker Representation for Attribute-Driven Privacy Preservation](https://mlsanity.com/paper/2012.04454/) (2020)
- [The Impact of Speech Anonymization on Pathology and Its Limits](https://mlsanity.com/paper/2404.08064/) (2024)
- [User-Level Differential Privacy against Attribute Inference Attack of Speech Emotion Recognition in Federated Learning](https://mlsanity.com/paper/2204.02500/) (2022)
- [The Privacy ZEBRA: Zero Evidence Biometric Recognition Assessment](https://mlsanity.com/paper/2005.09413/) (2020)
- [Asynchronous Voice Anonymization Using Adversarial Perturbation On Speaker Embedding](https://mlsanity.com/paper/2406.08200/) (2024)

## Built on by (most cited first, 1 of 1)

- [Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models](https://mlsanity.com/paper/2607.16870/) (2026)

## Abstract (arXiv)

Voice anonymization aims to conceal speaker identity and attributes while preserving intelligibility, but current evaluations rely almost exclusively on Equal Error Rate (EER) that obscures whether adversaries can mount high-precision attacks. We argue that privacy should instead be evaluated in the low false-positive rate (FPR) regime, where even a small number of successful identifications constitutes a meaningful breach. To this end, we introduce VoxGuard, a framework grounded in differential privacy and membership inference that formalizes two complementary notions: User Privacy, preventing speaker re-identification, and Attribute Privacy, protecting sensitive traits such as gender and accent. Across synthetic and real datasets, we find that informed adversaries, especially those using fine-tuned models and max-similarity scoring, achieve orders-of-magnitude stronger attacks at low-FPR despite similar EER. For attributes, we show that simple transparent attacks recover gender and accent with near-perfect accuracy even after anonymization. Our results demonstrate that EER substantially underestimates leakage, highlighting the need for low-FPR evaluation, and recommend VoxGuard as a benchmark for evaluating privacy leakage.