# Anonymization, Not Elimination: Utility-Preserved Speech Anonymization

arXiv: https://arxiv.org/abs/2604.17000 · HTML: https://mlsanity.com/paper/2604.17000/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Builds on (most cited first, 30 of 34)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Speaker Anonymization Using X-vector and Neural Waveform Models](https://mlsanity.com/paper/1905.13561/) (2019)
- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music](https://mlsanity.com/paper/2306.15412/) (2023)
- [Design Choices for X-vector Based Speaker Anonymization](https://mlsanity.com/paper/2005.08601/) (2020)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques](https://mlsanity.com/paper/2308.04455/) (2023)
- [Speaker anonymization using neural audio codec language models](https://mlsanity.com/paper/2309.14129/) (2023)
- [Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy](https://mlsanity.com/paper/2210.07002/) (2022)
- [The GDPR & Speech Data: Reflections of Legal and Technology Communities, First Steps towards a Common Understanding](https://mlsanity.com/paper/1907.03458/) (2019)
- [Speaker anonymization using orthogonal Householder neural network](https://mlsanity.com/paper/2305.18823/) (2023)
- [The Impact of Speech Anonymization on Pathology and Its Limits](https://mlsanity.com/paper/2404.08064/) (2024)
- [Are disentangled representations all you need to build speaker anonymization systems?](https://mlsanity.com/paper/2208.10497/) (2022)
- [MUSA: Multi-lingual Speaker Anonymization via Serial Disentanglement](https://mlsanity.com/paper/2407.11629/) (2024)
- [SynVox2: Towards a privacy-friendly VoxCeleb2 dataset](https://mlsanity.com/paper/2309.06141/) (2023)
- [Privacy in Speech Technology](https://mlsanity.com/paper/2305.05227/) (2023)
- [Exploratory Evaluation of Speech Content Masking](https://mlsanity.com/paper/2401.03936/) (2024)

## Built on by (most cited first, 1 of 1)

- [AegisVox: Factorized Privacy and Clinical Preservation for Long-Form Speech](https://mlsanity.com/paper/2610.14643/) (2026)

## Abstract (arXiv)

The growing reliance on large-scale speech data has made privacy protection a critical concern. However, existing anonymization approaches often degrade data utility, for example by disrupting acoustic continuity or reducing vocal diversity, which compromises the value of speech data for downstream tasks such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Speech Emotion Recognition (SER). Current evaluation practices are also limited, as they mainly rely on direct testing of anonymized speech with pretrained models, providing only a partial view of utility. To address these issues, we propose a novel two-stage framework that protects both linguistic content and acoustic identity while maintaining usability. For content privacy, we employ a generative speech editing model to seamlessly replace personally identifiable information (PII), and for voice privacy, we introduce F3-VA, a flow-matching-based anonymization framework with a three-stage design that produces diverse and distinct anonymized speakers. To enable a more comprehensive assessment, we evaluate privacy using both acoustic- and content-based speaker verification metrics, and assess utility by training ASR, TTS, and SER models from scratch. Experimental results show that our framework achieves stronger privacy protection with minimal utility degradation compared to baselines from the VoicePrivacy Challenge, while the proposed evaluation protocol provides a more realistic reflection of the utility of anonymized speech under privacy protection.