# Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network

arXiv: https://arxiv.org/abs/2106.07229 · HTML: https://mlsanity.com/paper/2106.07229/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 47 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [CryptoDL: Deep Neural Networks over Encrypted Data](https://mlsanity.com/paper/1711.05189/) (2017)
- [Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference](https://mlsanity.com/paper/1811.09953/) (2018)
- [SHE: A Fast and Accurate Deep Neural Network for Encrypted Data](https://mlsanity.com/paper/1906.00148/) (2019)
- [Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data](https://mlsanity.com/paper/2105.10879/) (2021)
- [nGraph-HE: A Graph Compiler for Deep Learning on Homomorphically Encrypted Data](https://mlsanity.com/paper/1810.10121/) (2018)
- [nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data](https://mlsanity.com/paper/1908.04172/) (2019)
- [Towards the AlexNet Moment for Homomorphic Encryption: HCNN, theFirst Homomorphic CNN on Encrypted Data with GPUs](https://mlsanity.com/paper/1811.00778/) (2018)
- [SEALion: a Framework for Neural Network Inference on Encrypted Data](https://mlsanity.com/paper/1904.12840/) (2019)

## Built on by (most cited first, 30 of 57)

- [Re-thinking Model Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/2304.01669/) (2023)
- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE](https://mlsanity.com/paper/2310.08012/) (2023)
- [SLIP: Securing LLMs IP Using Weights Decomposition](https://mlsanity.com/paper/2407.10886/) (2024)
- [HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption](https://mlsanity.com/paper/2403.14111/) (2024)
- [Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learning](https://mlsanity.com/paper/2306.00127/) (2023)
- [A methodology for training homomorphicencryption friendly neural networks](https://mlsanity.com/paper/2111.03362/) (2021)
- [HyPHEN: A Hybrid Packing Method and Optimizations for Homomorphic Encryption-Based Neural Networks](https://mlsanity.com/paper/2302.02407/) (2023)
- [Fast-FedUL: A Training-Free Federated Unlearning with Provable Skew Resilience](https://mlsanity.com/paper/2405.18040/) (2024)
- [Private and Reliable Neural Network Inference](https://mlsanity.com/paper/2210.15614/) (2022)
- [Catch-Only-One: Non-Transferable Examples for Model-Specific Authorization](https://mlsanity.com/paper/2510.10982/) (2025)
- [Fully Homomorphically Encrypted Deep Learning as a Service](https://mlsanity.com/paper/2107.12997/) (2021)
- [SVIP: Towards Verifiable Inference of Open-source Large Language Models](https://mlsanity.com/paper/2410.22307/) (2024)
- [Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption](https://mlsanity.com/paper/2304.14836/) (2023)
- [Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryption](https://mlsanity.com/paper/2310.10349/) (2023)
- [All Rivers Run to the Sea: Private Learning with Asymmetric Flows](https://mlsanity.com/paper/2312.05264/) (2023)
- [Fortify Your Foundations: Practical Privacy and Security for Foundation Model Deployments In The Cloud](https://mlsanity.com/paper/2410.05930/) (2024)
- [Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Survey](https://mlsanity.com/paper/2310.10665/) (2023)
- [From Principle to Practice: Vertical Data Minimization for Machine Learning](https://mlsanity.com/paper/2311.10500/) (2023)
- [A Scalable Multi-GPU Framework for Encrypted Large-Model Inference](https://mlsanity.com/paper/2512.11269/) (2025)
- [Robust Representation Learning for Privacy-Preserving Machine Learning: A Multi-Objective Autoencoder Approach](https://mlsanity.com/paper/2309.04427/) (2023)
- [Sharing is CAIRing: Characterizing Principles and Assessing Properties of Universal Privacy Evaluation for Synthetic Tabular Data](https://mlsanity.com/paper/2312.12216/) (2023)
- [MTRGL:Effective Temporal Correlation Discerning through Multi-modal Temporal Relational Graph Learning](https://mlsanity.com/paper/2401.14199/) (2024)
- [HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference](https://mlsanity.com/paper/2401.15970/) (2024)
- [Unveiling Hidden Visual Information: A Reconstruction Attack Against Adversarial Visual Information Hiding](https://mlsanity.com/paper/2408.04261/) (2024)
- [FuSeFL: Fully Secure and Scalable Federated Learning](https://mlsanity.com/paper/2507.13591/) (2025)
- [EDLaaS: Fully Homomorphic Encryption Over Neural Network Graphs for Vision and Private Strawberry Yield Forecasting](https://mlsanity.com/paper/2110.13638/) (2021)
- [On Achieving Privacy-Preserving State-of-the-Art Edge Intelligence](https://mlsanity.com/paper/2302.05323/) (2023)
- [Dynamic Fair Federated Learning Based on Reinforcement Learning](https://mlsanity.com/paper/2311.00959/) (2023)

## Abstract (arXiv)

Fully homomorphic encryption (FHE) is one of the prospective tools for privacypreserving machine learning (PPML), and several PPML models have been proposed based on various FHE schemes and approaches. Although the FHE schemes are known as suitable tools to implement PPML models, previous PPML models on FHE encrypted data are limited to only simple and non-standard types of machine learning models. These non-standard machine learning models are not proven efficient and accurate with more practical and advanced datasets. Previous PPML schemes replace non-arithmetic activation functions with simple arithmetic functions instead of adopting approximation methods and do not use bootstrapping, which enables continuous homomorphic evaluations. Thus, they could not use standard activation functions and could not employ a large number of layers. The maximum classification accuracy of the existing PPML model with the FHE for the CIFAR-10 dataset was only 77% until now. In this work, we firstly implement the standard ResNet-20 model with the RNS-CKKS FHE with bootstrapping and verify the implemented model with the CIFAR-10 dataset and the plaintext model parameters. Instead of replacing the non-arithmetic functions with the simple arithmetic function, we use state-of-the-art approximation methods to evaluate these non-arithmetic functions, such as the ReLU, with sufficient precision [1]. Further, for the first time, we use the bootstrapping technique of the RNS-CKKS scheme in the proposed model, which enables us to evaluate a deep learning model on the encrypted data. We numerically verify that the proposed model with the CIFAR-10 dataset shows 98.67% identical results to the original ResNet-20 model with non-encrypted data. The classification accuracy of the proposed model is 90.67%, which is pretty close to that of the original ResNet-20 CNN model...