# Configurable Privacy-Preserving Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2104.00766 · HTML: https://mlsanity.com/paper/2104.00766/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 27 of 27)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [Introducing the VoicePrivacy Initiative](https://mlsanity.com/paper/2005.01387/) (2020)
- [SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systems](https://mlsanity.com/paper/2007.06622/) (2020)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Evaluating Voice Conversion-based Privacy Protection against Informed Attackers](https://mlsanity.com/paper/1911.03934/) (2019)
- [Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends](https://mlsanity.com/paper/2001.00378/) (2020)
- [Design Choices for X-vector Based Speaker Anonymization](https://mlsanity.com/paper/2005.08601/) (2020)
- [Personal VAD: Speaker-Conditioned Voice Activity Detection](https://mlsanity.com/paper/1908.04284/) (2019)
- [Privacy-preserving Voice Analysis via Disentangled Representations](https://mlsanity.com/paper/2007.15064/) (2020)
- [Privacy Enhanced Multimodal Neural Representations for Emotion Recognition](https://mlsanity.com/paper/1910.13212/) (2019)
- [Generative Speech Coding with Predictive Variance Regularization](https://mlsanity.com/paper/2102.09660/) (2021)
- [Preech: A System for Privacy-Preserving Speech Transcription](https://mlsanity.com/paper/1909.04198/) (2019)
- [Online End-to-End Neural Diarization with Speaker-Tracing Buffer](https://mlsanity.com/paper/2006.02616/) (2020)
- [A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings](https://mlsanity.com/paper/2012.02221/) (2020)
- [crank: An Open-Source Software for Nonparallel Voice Conversion Based on Vector-Quantized Variational Autoencoder](https://mlsanity.com/paper/2103.02858/) (2021)
- [Paralinguistic Privacy Protection at the Edge](https://mlsanity.com/paper/2011.02930/) (2020)
- [Emotion Filtering at the Edge](https://mlsanity.com/paper/1909.08500/) (2019)

## Built on by (most cited first, 7 of 7)

- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Protecting Bystander Privacy via Selective Hearing in Audio LLMs](https://mlsanity.com/paper/2512.06380/) (2025)
- [Representation Learning for Audio Privacy Preservation using Source Separation and Robust Adversarial Learning](https://mlsanity.com/paper/2308.04960/) (2023)
- [Privacy-Preserving Speech Representation Learning using Vector Quantization](https://mlsanity.com/paper/2203.09518/) (2022)
- [Privacy-Preserving End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2403.15510/) (2024)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)
- [EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis](https://mlsanity.com/paper/2606.24941/) (2026)

## Abstract (arXiv)

Voice assistive technologies have given rise to far-reaching privacy and security concerns. In this paper we investigate whether modular automatic speech recognition (ASR) can improve privacy in voice assistive systems by combining independently trained separation, recognition, and discretization modules to design configurable privacy-preserving ASR systems. We evaluate privacy concerns and the effects of applying various state-of-the-art techniques at each stage of the system, and report results using task-specific metrics (i.e. WER, ABX, and accuracy). We show that overlapping speech inputs to ASR systems present further privacy concerns, and how these may be mitigated using speech separation and optimization techniques. Our discretization module is shown to minimize paralinguistics privacy leakage from ASR acoustic models to levels commensurate with random guessing. We show that voice privacy can be configurable, and argue this presents new opportunities for privacy-preserving applications incorporating ASR.