# Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification

arXiv: https://arxiv.org/abs/2609.18673 · HTML: https://mlsanity.com/paper/2609.18673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context](https://mlsanity.com/paper/2110.04410/) (2021)
- [Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST](https://mlsanity.com/paper/2509.14128/) (2025)
- [ECAPA-TDNN Embeddings for Speaker Diarization](https://mlsanity.com/paper/2104.01466/) (2021)
- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [PASS: Protected Attribute Suppression System for Mitigating Bias in Face Recognition](https://mlsanity.com/paper/2108.03764/) (2021)
- [The VoicePrivacy 2022 Challenge: Progress and Perspectives in Voice Anonymisation](https://mlsanity.com/paper/2407.11516/) (2024)
- [Deep Convolutional Neural Networks in the Face of Caricature: Identity and Image Revealed](https://mlsanity.com/paper/1812.10902/) (2018)
- [Deep Convolutional Neural Network Features and the Original Image](https://mlsanity.com/paper/1611.01751/) (2016)
- [Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization](https://mlsanity.com/paper/2412.17164/) (2024)
- [Attacking Voice Anonymization Systems with Augmented Feature and Speaker Identity Difference](https://mlsanity.com/paper/2412.19068/) (2024)
- [SpecWav-Attack: Leveraging Spectrogram Resizing and Wav2Vec 2.0 for Attacking Anonymized Speech](https://mlsanity.com/paper/2505.09616/) (2025)
- [Demographic Attributes Prediction from Speech Using WavLM Embeddings](https://mlsanity.com/paper/2502.12007/) (2025)
- [Benchmark Dataset Dynamics, Bias and Privacy Challenges in Voice Biometrics Research](https://mlsanity.com/paper/2304.03858/) (2023)
- [Toward Fair Speech Technologies: A Comprehensive Survey of Bias and Fairness in Speech AI](https://mlsanity.com/paper/2605.01597/) (2026)
- [Any-to-any Speaker Attribute Perturbation for Asynchronous Voice Anonymization](https://mlsanity.com/paper/2508.15565/) (2025)
- [Measuring Soft Biometric Leakage in Speaker De-Identification Systems](https://mlsanity.com/paper/2509.14469/) (2025)

## Abstract (arXiv)

Speaker de-identification (SDID) aims to preserve privacy by concealing speaker identity while maintaining speech utility. However, current evaluations often reduce privacy to a single dimension - biometric verification performance - typically measured by Equal Error Rate (EER). This narrow focus ignores critical leakage channels, such as soft biometric inference, embedding-level re-identification, and structural template similarity, which threaten the unlinkability and irreversibility of biometric references. We propose a holistic evaluation framework across five complementary metrics: (i) EER, (ii) soft biometric leakage score , (iii) cumulative match characteristic re-identification analysis, (iv) canonical correlation analysis and Procrustes embedding alignment, and (v) intelligibility via word error rate and semantic similarity. Evaluating five SDID systems from the IARPA ARTS program, we demonstrate that these metrics capture independent dimensions of information leakage. Our results indicate that reliance on a single metric can misrepresent the privacy properties of an SDID system.