# Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?

arXiv: https://arxiv.org/abs/1911.04913 · HTML: https://mlsanity.com/paper/1911.04913/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 22 of 22)

- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities](https://mlsanity.com/paper/2102.04257/) (2021)
- [Differentially Private Speaker Anonymization](https://mlsanity.com/paper/2202.11823/) (2022)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Disentangling Voice and Content with Self-Supervision for Speaker Recognition](https://mlsanity.com/paper/2310.01128/) (2023)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)
- [Protecting gender and identity with disentangled speech representations](https://mlsanity.com/paper/2104.11051/) (2021)
- [FedNST: Federated Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2206.02797/) (2022)
- [Generating gender-ambiguous voices for privacy-preserving speech recognition](https://mlsanity.com/paper/2207.01052/) (2022)
- [Introducing Model Inversion Attacks on Automatic Speaker Recognition](https://mlsanity.com/paper/2301.03206/) (2023)
- [HearSay Benchmark: Do Audio LLMs Leak What They Hear?](https://mlsanity.com/paper/2601.03783/) (2026)
- [Adversarial Representation Learning for Robust Privacy Preservation in Audio](https://mlsanity.com/paper/2305.00011/) (2023)
- [An Attribute-Aligned Strategy for Learning Speech Representation](https://mlsanity.com/paper/2106.02810/) (2021)
- [Evaluating X-vector-based Speaker Anonymization under White-box Assessment](https://mlsanity.com/paper/2109.11946/) (2021)
- [Self-Supervised Speech Representations Preserve Speech Characteristics while Anonymizing Voices](https://mlsanity.com/paper/2204.01677/) (2022)
- [Representation Learning for Audio Privacy Preservation using Source Separation and Robust Adversarial Learning](https://mlsanity.com/paper/2308.04960/) (2023)
- [Privacy-Preserving Speech Representation Learning using Vector Quantization](https://mlsanity.com/paper/2203.09518/) (2022)
- [Privacy-Utility Balanced Voice De-Identification Using Adversarial Examples](https://mlsanity.com/paper/2211.05446/) (2022)
- [Long-term Conversation Analysis: Exploring Utility and Privacy](https://mlsanity.com/paper/2306.16071/) (2023)
- [Privacy-Preserving End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2403.15510/) (2024)
- [Learning Private Representations through Entropy-based Adversarial Training](https://mlsanity.com/paper/2507.10194/) (2025)
- [Discriminating real and synthetic super-resolved audio samples using embedding-based classifiers](https://mlsanity.com/paper/2601.03443/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) is a key technology in many services and applications. This typically requires user devices to send their speech data to the cloud for ASR decoding. As the speech signal carries a lot of information about the speaker, this raises serious privacy concerns. As a solution, an encoder may reside on each user device which performs local computations to anonymize the representation. In this paper, we focus on the protection of speaker identity and study the extent to which users can be recognized based on the encoded representation of their speech as obtained by a deep encoder-decoder architecture trained for ASR. Through speaker identification and verification experiments on the Librispeech corpus with open and closed sets of speakers, we show that the representations obtained from a standard architecture still carry a lot of information about speaker identity. We then propose to use adversarial training to learn representations that perform well in ASR while hiding speaker identity. Our results demonstrate that adversarial training dramatically reduces the closed-set classification accuracy, but this does not translate into increased open-set verification error hence into increased protection of the speaker identity in practice. We suggest several possible reasons behind this negative result.