# HiddenSpeaker: Generate Imperceptible Unlearnable Audios for Speaker Verification System

arXiv: https://arxiv.org/abs/2405.15655 · HTML: https://mlsanity.com/paper/2405.15655/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 22 of 22)

- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Fast Yet Effective Machine Unlearning](https://mlsanity.com/paper/2111.08947/) (2021)
- [Unlearnable Examples: Making Personal Data Unexploitable](https://mlsanity.com/paper/2101.04898/) (2021)
- [Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses](https://mlsanity.com/paper/2012.10544/) (2020)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Speaker Recognition Based on Deep Learning: An Overview](https://mlsanity.com/paper/2012.00931/) (2020)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [MFA: TDNN with Multi-scale Frequency-channel Attention for Text-independent Speaker Verification with Short Utterances](https://mlsanity.com/paper/2202.01624/) (2022)
- [Improving fairness in speaker verification via Group-adapted Fusion Network](https://mlsanity.com/paper/2202.11323/) (2022)
- [InfoScrub: Towards Attribute Privacy by Targeted Obfuscation](https://mlsanity.com/paper/2005.10329/) (2020)
- [MASTERKEY: Practical Backdoor Attack Against Speaker Verification Systems](https://mlsanity.com/paper/2309.06981/) (2023)
- [VenoMave: Targeted Poisoning Against Speech Recognition](https://mlsanity.com/paper/2010.10682/) (2020)
- [Utilizing Noise Addition for Data Privacy, an Overview](https://mlsanity.com/paper/1309.3958/) (2013)
- [Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?](https://mlsanity.com/paper/2306.15268/) (2023)
- [WaveFuzz: A Clean-Label Poisoning Attack to Protect Your Voice](https://mlsanity.com/paper/2203.13497/) (2022)
- [A Speech Representation Anonymization Framework via Selective Noise Perturbation](https://mlsanity.com/paper/2203.14171/) (2022)
- [DS-TDNN: Dual-stream Time-delay Neural Network with Global-aware Filter for Speaker Verification](https://mlsanity.com/paper/2303.11020/) (2023)

## Built on by (most cited first, 4 of 4)

- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [A Survey on Unlearnable Data](https://mlsanity.com/paper/2503.23536/) (2025)
- [Towards Operationalizing Right to Data Protection](https://mlsanity.com/paper/2411.08506/) (2024)
- [E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis](https://mlsanity.com/paper/2511.07099/) (2025)

## Abstract (arXiv)

In recent years, the remarkable advancements in deep neural networks have brought tremendous convenience. However, the training process of a highly effective model necessitates a substantial quantity of samples, which brings huge potential threats, like unauthorized exploitation with privacy leakage. In response, we propose a framework named HiddenSpeaker, embedding imperceptible perturbations within the training speech samples and rendering them unlearnable for deep-learning-based speaker verification systems that employ large-scale speakers for efficient training. The HiddenSpeaker utilizes a simplified error-minimizing method named Single-Level Error-Minimizing (SLEM) to generate specific and effective perturbations. Additionally, a hybrid objective function is employed for human perceptual optimization, ensuring the perturbation is indistinguishable from human listeners. We conduct extensive experiments on multiple state-of-the-art (SOTA) models in the speaker verification domain to evaluate HiddenSpeaker. Our results demonstrate that HiddenSpeaker not only deceives the model with unlearnable samples but also enhances the imperceptibility of the perturbations, showcasing strong transferability across different models.