# Audit and Improve Robustness of Private Neural Networks on Encrypted Data

arXiv: https://arxiv.org/abs/2209.09996 · HTML: https://mlsanity.com/paper/2209.09996/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers](https://mlsanity.com/paper/1906.04584/) (2019)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors](https://mlsanity.com/paper/1807.07978/) (2018)
- [Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack](https://mlsanity.com/paper/1811.09310/) (2018)
- [Sign-OPT: A Query-Efficient Hard-label Adversarial Attack](https://mlsanity.com/paper/1909.10773/) (2019)
- [Adversarial Examples that Fool both Computer Vision and Time-Limited Humans](https://mlsanity.com/paper/1802.08195/) (2018)
- [Low Latency Privacy Preserving Inference](https://mlsanity.com/paper/1812.10659/) (2018)
- [Stateful Detection of Black-Box Adversarial Attacks](https://mlsanity.com/paper/1907.05587/) (2019)
- [Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization](https://mlsanity.com/paper/1905.06635/) (2019)
- [Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?](https://mlsanity.com/paper/2203.08392/) (2022)
- [Random Noise Defense Against Query-Based Black-Box Attacks](https://mlsanity.com/paper/2104.11470/) (2021)
- [Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference](https://mlsanity.com/paper/1811.09953/) (2018)
- [AdvMind: Inferring Adversary Intent of Black-Box Attacks](https://mlsanity.com/paper/2006.09539/) (2020)

## Built on by (most cited first, 4 of 4)

- [TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models](https://mlsanity.com/paper/2306.06815/) (2023)
- [TFHE-Coder: Evaluating LLM-agentic Fully Homomorphic Encryption Code Generation](https://mlsanity.com/paper/2503.12217/) (2025)
- [CryptoTrain: Fast Secure Training on Encrypted Dataset](https://mlsanity.com/paper/2409.16675/) (2024)
- [DictPFL: Efficient and Private Federated Learning on Encrypted Gradients](https://mlsanity.com/paper/2510.21086/) (2025)

## Abstract (arXiv)

Performing neural network inference on encrypted data without decryption is one popular method to enable privacy-preserving neural networks (PNet) as a service. Compared with regular neural networks deployed for machine-learning-as-a-service, PNet requires additional encoding, e.g., quantized-precision numbers, and polynomial activation. Encrypted input also introduces novel challenges such as adversarial robustness and security. To the best of our knowledge, we are the first to study questions including (i) Whether PNet is more robust against adversarial inputs than regular neural networks? (ii) How to design a robust PNet given the encrypted input without decryption? We propose PNet-Attack to generate black-box adversarial examples that can successfully attack PNet in both target and untarget manners. The attack results show that PNet robustness against adversarial inputs needs to be improved. This is not a trivial task because the PNet model owner does not have access to the plaintext of the input values, which prevents the application of existing detection and defense methods such as input tuning, model normalization, and adversarial training. To tackle this challenge, we propose a new fast and accurate noise insertion method, called RPNet, to design Robust and Private Neural Networks. Our comprehensive experiments show that PNet-Attack reduces at least 2.5× queries than prior works. We theoretically analyze our RPNet methods and demonstrate that RPNet can decrease ~ 91.88% attack success rate.