# Orion: A Fully Homomorphic Encryption Framework for Deep Learning

arXiv: https://arxiv.org/abs/2311.03470 · HTML: https://mlsanity.com/paper/2311.03470/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2023.

## Built on by (most cited first, 15 of 15)

- [Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs](https://mlsanity.com/paper/2509.18886/) (2025)
- [Leveraging ASIC AI Chips for Homomorphic Encryption](https://mlsanity.com/paper/2501.07047/) (2025)
- [A Scalable Multi-GPU Framework for Encrypted Large-Model Inference](https://mlsanity.com/paper/2512.11269/) (2025)
- [AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems](https://mlsanity.com/paper/2604.03425/) (2026)
- [ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour](https://mlsanity.com/paper/2607.23478/) (2026)
- [CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation](https://mlsanity.com/paper/2511.19711/) (2025)
- [CHEHAB RL: Learning to Optimize Fully Homomorphic Encryption Computations](https://mlsanity.com/paper/2601.19367/) (2026)
- [Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption](https://mlsanity.com/paper/2509.01253/) (2025)
- [FHE-Agent: Automating CKKS Configuration for Practical Encrypted Inference via an LLM-Guided Agentic Framework](https://mlsanity.com/paper/2511.18653/) (2025)
- [Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks](https://mlsanity.com/paper/2604.16834/) (2026)
- [Encrypted Neural Networks without Overflows](https://mlsanity.com/paper/2605.23096/) (2026)
- [FEnc2: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding](https://mlsanity.com/paper/2606.16359/) (2026)
- [HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation](https://mlsanity.com/paper/2609.01730/) (2026)
- [Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights](https://mlsanity.com/paper/2609.35352/) (2026)
- [Static Bootstrap Placement for Encrypted Language Model Decoding](https://mlsanity.com/paper/2610.04912/) (2026)

## Abstract (arXiv)

Fully Homomorphic Encryption (FHE) has the potential to substantially improve privacy and security by enabling computation directly on encrypted data. This is especially true with deep learning, as today, many popular user services are powered by neural networks in the cloud. Beyond its well-known high computational costs, one of the major challenges facing wide-scale deployment of FHE-secured neural inference is effectively mapping these networks to FHE primitives. FHE poses many programming challenges including packing large vectors, managing accumulated noise, and translating arbitrary and general-purpose programs to the limited instruction set provided by FHE. These challenges make building large FHE neural networks intractable using the tools available today. In this paper we address these challenges with Orion, a fully-automated framework for private neural inference using FHE. Orion accepts deep neural networks written in PyTorch and translates them into efficient FHE programs. We achieve this by proposing a novel single-shot multiplexed packing strategy for arbitrary convolutions and through a new, efficient technique to automate bootstrap placement and scale management. We evaluate Orion on common benchmarks used by the FHE deep learning community and outperform state-of-the-art by 2.38x on ResNet-20, the largest network they report. Orion's techniques enable processing much deeper and larger networks. We demonstrate this by evaluating ResNet-50 on ImageNet and present the first high-resolution FHE object detection experiments using a YOLO-v1 model with 139 million parameters. Orion is open-source for all to use at: https://github.com/baahl-nyu/orion