# Encrypted Speech Recognition using Deep Polynomial Networks

arXiv: https://arxiv.org/abs/1905.05605 · HTML: https://mlsanity.com/paper/1905.05605/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2019.

## Built on by (most cited first, 8 of 8)

- [The VoicePrivacy 2020 Challenge: Results and findings](https://mlsanity.com/paper/2109.00648/) (2021)
- [Differentially Private Speaker Anonymization](https://mlsanity.com/paper/2202.11823/) (2022)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [SoK: Privacy-preserving Deep Learning with Homomorphic Encryption](https://mlsanity.com/paper/2112.12855/) (2021)
- [Adversarial Audio Synthesis with Complex-valued Polynomial Networks](https://mlsanity.com/paper/2206.06811/) (2022)
- [PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding](https://mlsanity.com/paper/2602.01322/) (2026)
- [Mitigating Closed-model Adversarial Examples with Bayesian Neural Modeling for Enhanced End-to-End Speech Recognition](https://mlsanity.com/paper/2202.08532/) (2022)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)

## Abstract (arXiv)

The cloud-based speech recognition/API provides developers or enterprises an easy way to create speech-enabled features in their applications. However, sending audios about personal or company internal information to the cloud, raises concerns about the privacy and security issues. The recognition results generated in cloud may also reveal some sensitive information. This paper proposes a deep polynomial network (DPN) that can be applied to the encrypted speech as an acoustic model. It allows clients to send their data in an encrypted form to the cloud to ensure that their data remains confidential, at mean while the DPN can still make frame-level predictions over the encrypted speech and return them in encrypted form. One good property of the DPN is that it can be trained on unencrypted speech features in the traditional way. To keep the cloud away from the raw audio and recognition results, a cloud-local joint decoding framework is also proposed. We demonstrate the effectiveness of model and framework on the Switchboard and Cortana voice assistant tasks with small performance degradation and latency increased comparing with the traditional cloud-based DNNs.