# Privacy-Preserving End-to-End Spoken Language Understanding

arXiv: https://arxiv.org/abs/2403.15510 · HTML: https://mlsanity.com/paper/2403.15510/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [Deep Speaker: an End-to-End Neural Speaker Embedding System](https://mlsanity.com/paper/1705.02304/) (2017)
- [Privacy-Preserving Machine Learning: Methods, Challenges and Directions](https://mlsanity.com/paper/2108.04417/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [Privacy-preserving Voice Analysis via Disentangled Representations](https://mlsanity.com/paper/2007.15064/) (2020)
- [Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?](https://mlsanity.com/paper/1911.04913/) (2019)
- [Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model](https://mlsanity.com/paper/2112.07254/) (2021)
- [Adversarial Disentanglement of Speaker Representation for Attribute-Driven Privacy Preservation](https://mlsanity.com/paper/2012.04454/) (2020)
- [Configurable Privacy-Preserving Automatic Speech Recognition](https://mlsanity.com/paper/2104.00766/) (2021)
- [Paralinguistic Privacy Protection at the Edge](https://mlsanity.com/paper/2011.02930/) (2020)
- [Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks](https://mlsanity.com/paper/2101.08919/) (2021)
- [Protecting gender and identity with disentangled speech representations](https://mlsanity.com/paper/2104.11051/) (2021)

## Abstract (arXiv)

Spoken language understanding (SLU), one of the key enabling technologies for human-computer interaction in IoT devices, provides an easy-to-use user interface. Human speech can contain a lot of user-sensitive information, such as gender, identity, and sensitive content. New types of security and privacy breaches have thus emerged. Users do not want to expose their personal sensitive information to malicious attacks by untrusted third parties. Thus, the SLU system needs to ensure that a potential malicious attacker cannot deduce the sensitive attributes of the users, while it should avoid greatly compromising the SLU accuracy. To address the above challenge, this paper proposes a novel SLU multi-task privacy-preserving model to prevent both the speech recognition (ASR) and identity recognition (IR) attacks. The model uses the hidden layer separation technique so that SLU information is distributed only in a specific portion of the hidden layer, and the other two types of information are removed to obtain a privacy-secure hidden layer. In order to achieve good balance between efficiency and privacy, we introduce a new mechanism of model pre-training, namely joint adversarial training, to further enhance the user privacy. Experiments over two SLU datasets show that the proposed method can reduce the accuracy of both the ASR and IR attacks close to that of a random guess, while leaving the SLU performance largely unaffected.