# nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data

arXiv: https://arxiv.org/abs/1908.04172 · HTML: https://mlsanity.com/paper/1908.04172/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2019.

## Built on by (most cited first, 21 of 21)

- [Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network](https://mlsanity.com/paper/2106.07229/) (2021)
- [THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption](https://mlsanity.com/paper/2206.00216/) (2022)
- [TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption](https://mlsanity.com/paper/2104.03152/) (2021)
- [SIRNN: A Math Library for Secure RNN Inference](https://mlsanity.com/paper/2105.04236/) (2021)
- [A methodology for training homomorphicencryption friendly neural networks](https://mlsanity.com/paper/2111.03362/) (2021)
- [Enabling Homomorphically Encrypted Inference for Large DNN Models](https://mlsanity.com/paper/2103.16139/) (2021)
- [Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption](https://mlsanity.com/paper/2304.14836/) (2023)
- [Privacy-Preserving Diffusion Model Using Homomorphic Encryption](https://mlsanity.com/paper/2403.05794/) (2024)
- [A Training Framework for Optimal and Stable Training of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11589/) (2025)
- [Efficient Pruning for Machine Learning Under Homomorphic Encryption](https://mlsanity.com/paper/2207.03384/) (2022)
- [Partially Oblivious Neural Network Inference](https://mlsanity.com/paper/2210.15189/) (2022)
- [HE-MAN -- Homomorphically Encrypted MAchine learning with oNnx models](https://mlsanity.com/paper/2302.08260/) (2023)
- [Bi-CryptoNets: Leveraging Different-Level Privacy for Encrypted Inference](https://mlsanity.com/paper/2402.01296/) (2024)
- [ReBoot: Encrypted Training of Deep Neural Networks with CKKS Bootstrapping](https://mlsanity.com/paper/2506.19693/) (2025)
- [SEEK: model extraction attack against hybrid secure inference protocols](https://mlsanity.com/paper/2209.06373/) (2022)
- [Privacy-Preserving Credit Card Fraud Detection using Homomorphic Encryption](https://mlsanity.com/paper/2211.06675/) (2022)
- [Towards Automated Homomorphic Encryption Parameter Selection with Fuzzy Logic and Linear Programming](https://mlsanity.com/paper/2302.08930/) (2023)
- [Speed-up of Data Analysis with Kernel Trick in Encrypted Domain](https://mlsanity.com/paper/2406.09716/) (2024)
- [CBNN: 3-Party Secure Framework for Customized Binary Neural Networks Inference](https://mlsanity.com/paper/2412.16449/) (2024)
- [Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks](https://mlsanity.com/paper/2604.16834/) (2026)
- [Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference](https://mlsanity.com/paper/2605.22237/) (2026)

## Abstract (arXiv)

In previous work, Boemer et al. introduced nGraph-HE, an extension to the Intel nGraph deep learning (DL) compiler, that enables data scientists to deploy models with popular frameworks such as TensorFlow and PyTorch with minimal code changes. However, the class of supported models was limited to relatively shallow networks with polynomial activations. Here, we introduce nGraph-HE2, which extends nGraph-HE to enable privacy-preserving inference on standard, pre-trained models using their native activation functions and number fields (typically real numbers). The proposed framework leverages the CKKS scheme, whose support for real numbers is friendly to data science, and a client-aided model using a two-party approach to compute activation functions. We first present CKKS-specific optimizations, enabling a 3x-88x runtime speedup for scalar encoding, and doubling the throughput through a novel use of CKKS plaintext packing into complex numbers. Second, we optimize ciphertext-plaintext addition and multiplication, yielding 2.6x-4.2x runtime speedup. Third, we exploit two graph-level optimizations: lazy rescaling and depth-aware encoding, which allow us to significantly improve performance. Together, these optimizations enable state-of-the-art throughput of 1,998 images/s on the CryptoNets network. Using the client-aided model, we also present homomorphic evaluation of (to our knowledge) the largest network to date, namely, pre-trained MobileNetV2 models on the ImageNet dataset, with 60.4percent/82.7percent top-1/top-5 accuracy and an amortized runtime of 381 ms/image.