# Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data

arXiv: https://arxiv.org/abs/2105.10879 · HTML: https://mlsanity.com/paper/2105.10879/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2021.

## Built on by (most cited first, 27 of 27)

- [Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network](https://mlsanity.com/paper/2106.07229/) (2021)
- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE](https://mlsanity.com/paper/2310.08012/) (2023)
- [Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning](https://mlsanity.com/paper/2107.12342/) (2021)
- [AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference](https://mlsanity.com/paper/2201.06699/) (2022)
- [HyPHEN: A Hybrid Packing Method and Optimizations for Homomorphic Encryption-Based Neural Networks](https://mlsanity.com/paper/2302.02407/) (2023)
- [Power-Softmax: Towards Secure LLM Inference over Encrypted Data](https://mlsanity.com/paper/2410.09457/) (2024)
- [NeuJeans: Private Neural Network Inference with Joint Optimization of Convolution and FHE Bootstrapping](https://mlsanity.com/paper/2312.04356/) (2023)
- [Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption](https://mlsanity.com/paper/2304.14836/) (2023)
- [Fast and Private Inference of Deep Neural Networks by Co-designing Activation Functions](https://mlsanity.com/paper/2306.08538/) (2023)
- [Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryption](https://mlsanity.com/paper/2310.10349/) (2023)
- [A Training Framework for Optimal and Stable Training of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11589/) (2025)
- [Robust Representation Learning for Privacy-Preserving Machine Learning: A Multi-Objective Autoencoder Approach](https://mlsanity.com/paper/2309.04427/) (2023)
- [Exploiting Unstructured Sparsity in Fully Homomorphic Encrypted DNNs](https://mlsanity.com/paper/2503.09184/) (2025)
- [CryptoFace: End-to-End Encrypted Face Recognition](https://mlsanity.com/paper/2509.00332/) (2025)
- [ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour](https://mlsanity.com/paper/2607.23478/) (2026)
- [Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement](https://mlsanity.com/paper/2402.11224/) (2024)
- [Privacy-aware Berrut Approximated Coded Computing for Federated Learning](https://mlsanity.com/paper/2405.01704/) (2024)
- [Traveling Salesman-Based Token Ordering Improves Stability in Homomorphically Encrypted Language Models](https://mlsanity.com/paper/2510.12343/) (2025)
- [Peregrine: One-Shot Fine-Tuning for FHE Inference of General Deep CNNs](https://mlsanity.com/paper/2511.18976/) (2025)
- [An End-to-End Homomorphically Encrypted Neural Network](https://mlsanity.com/paper/2502.16176/) (2025)
- [PRISM: Privacy-preserving Inference System with Homomorphic Encryption and Modular Activation](https://mlsanity.com/paper/2511.07807/) (2025)
- [Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference](https://mlsanity.com/paper/2605.22237/) (2026)
- [Encrypted Neural Networks without Overflows](https://mlsanity.com/paper/2605.23096/) (2026)
- [Beyond Classification: Task-Dependent Learnability under Privacy-Motivated Image Transformations](https://mlsanity.com/paper/2608.27066/) (2026)
- [Privacy-Preserving GNN Inference on Financial Transaction Graphs via Homomorphic Encryption](https://mlsanity.com/paper/2610.18054/) (2026)

## Abstract (arXiv)

Homomorphic encryption is one of the representative solutions to privacy-preserving machine learning (PPML) classification enabling the server to classify private data of clients while guaranteeing privacy. This work focuses on PPML using word-wise fully homomorphic encryption (FHE). In order to implement deep learning on word-wise homomorphic encryption (HE), the ReLU and max-pooling functions should be approximated by some polynomials for homomorphic operations. Most of the previous studies focus on HE-friendly networks, where the ReLU and max-pooling functions are approximated using low-degree polynomials. However, for the classification of the CIFAR-10 dataset, using a low-degree polynomial requires designing a new deep learning model and training. In addition, this approximation by low-degree polynomials cannot support deeper neural networks due to large approximation errors. Thus, we propose a precise polynomial approximation technique for the ReLU and max-pooling functions. Precise approximation using a single polynomial requires an exponentially high-degree polynomial, which results in a significant number of non-scalar multiplications. Thus, we propose a method to approximate the ReLU and max-pooling functions accurately using a composition of minimax approximate polynomials of small degrees. If we replace the ReLU and max-pooling functions with the proposed approximate polynomials, well-studied deep learning models such as ResNet and VGGNet can still be used without further modification for PPML on FHE. Even pre-trained parameters can be used without retraining. We approximate the ReLU and max-pooling functions in the ResNet-152 using the composition of minimax approximate polynomials of degrees 15, 27, and 29. Then, we succeed in classifying the plaintext ImageNet dataset with 77.52% accuracy, which is very close to the original model accuracy of 78.31%.