# Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics

arXiv: https://arxiv.org/abs/2610.08831 · HTML: https://mlsanity.com/paper/2610.08831/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Evaluating Voice Conversion-based Privacy Protection against Informed Attackers](https://mlsanity.com/paper/1911.03934/) (2019)
- [The GDPR & Speech Data: Reflections of Legal and Technology Communities, First Steps towards a Common Understanding](https://mlsanity.com/paper/1907.03458/) (2019)
- [Audio De-identification: A New Entity Recognition Task](https://mlsanity.com/paper/1903.07037/) (2019)
- [Preech: A System for Privacy-Preserving Speech Transcription](https://mlsanity.com/paper/1909.04198/) (2019)
- [H_eval: A new hybrid evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2211.01722/) (2022)
- [Revisiting Speech Content Privacy](https://mlsanity.com/paper/2110.06760/) (2021)
- [Exploratory Evaluation of Speech Content Masking](https://mlsanity.com/paper/2401.03936/) (2024)
- [Revisiting the Privacy of Low-Frequency Speech Signals: Exploring Resampling Methods, Evaluation Scenarios, and Speaker Characteristics](https://mlsanity.com/paper/2508.02483/) (2025)

## Abstract (arXiv)

As the use of smart devices continues to increase, their potential to capture sensitive speech content raises growing privacy concerns. It is therefore critical to develop techniques that prevent information leakage while preserving the utility of the audio, and evaluation metrics that accurately quantify the level of privacy without overestimating it. In this work, we evaluate the effectiveness of two obfuscation techniques in protecting speech content, with particular emphasis on named entities, by adapting entity-aware privacy metrics from the Natural Language Processing field to the speech privacy domain. Further, we investigate several attack scenarios and show that fine-tuning on entity-rich data improves attack performance for some entity categories but not others. Finally, we provide guidance on metric selection based on whether the obfuscation method preserves temporal alignment.