# Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks

arXiv: https://arxiv.org/abs/2101.08919 · HTML: https://mlsanity.com/paper/2101.08919/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2021.

## Built on by (most cited first, 5 of 5)

- [RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding](https://mlsanity.com/paper/2506.10289/) (2025)
- [Safeguarding Privacy in Edge Speech Understanding with Tiny Foundation Models](https://mlsanity.com/paper/2502.01649/) (2025)
- [Writing in The Air: Unconstrained Text Recognition from Finger Movement Using Spatio-Temporal Convolution](https://mlsanity.com/paper/2104.09021/) (2021)
- [Beyond Neural-on-Neural Approaches to Speaker Gender Protection](https://mlsanity.com/paper/2306.17700/) (2023)
- [Privacy-Preserving End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2403.15510/) (2024)

## Abstract (arXiv)

As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private. Existing solutions rely either on server-side methods or focus on hiding speaker identity. While these approaches reduce certain security concerns, they do not give users client-side control over whether their biometric information is sent to the server. In this paper, we formally define client-side privacy and discuss its three unique technical challenges: (1) direct manipulation of raw data on client devices, (2) adaptability with a broad range of server-side processing models, and (3) low time and space complexity for compatibility with limited-bandwidth devices. Solving these challenges requires new models that achieve high-fidelity reconstruction, privacy preservation of sensitive personal attributes, and efficiency during training and inference. As a step towards client-side privacy for speech recognition, we investigate three techniques spanning signal processing, disentangled representation learning, and adversarial training. Through a series of gender and accent masking tasks, we observe that each method has its unique strengths, but none manage to effectively balance the trade-offs between performance, privacy, and complexity. These insights call for more research in client-side privacy to ensure a safer deployment of cloud-based speech processing services.