# A Scalable Multi-GPU Framework for Encrypted Large-Model Inference

arXiv: https://arxiv.org/abs/2512.11269 · HTML: https://mlsanity.com/paper/2512.11269/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 12 Dec 2025.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network](https://mlsanity.com/paper/2106.07229/) (2021)
- [F1: A Fast and Programmable Accelerator for Fully Homomorphic Encryption (Extended Version)](https://mlsanity.com/paper/2109.05371/) (2021)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [Orion: A Fully Homomorphic Encryption Framework for Deep Learning](https://mlsanity.com/paper/2311.03470/) (2023)
- [ARK: Fully Homomorphic Encryption Accelerator with Runtime Data Generation and Inter-Operation Key Reuse](https://mlsanity.com/paper/2205.00922/) (2022)
- [TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU](https://mlsanity.com/paper/2212.14191/) (2022)
- [Enabling Homomorphically Encrypted Inference for Large DNN Models](https://mlsanity.com/paper/2103.16139/) (2021)
- [GME: GPU-based Microarchitectural Extensions to Accelerate Homomorphic Encryption](https://mlsanity.com/paper/2309.11001/) (2023)
- [Cheddar: A Swift Fully Homomorphic Encryption Library Designed for GPU Architectures](https://mlsanity.com/paper/2407.13055/) (2024)
- [HEIR: A Universal Compiler for Homomorphic Encryption](https://mlsanity.com/paper/2508.11095/) (2025)
- [Leveraging ASIC AI Chips for Homomorphic Encryption](https://mlsanity.com/paper/2501.07047/) (2025)
- [Recency Biased Causal Attention for Time-series Forecasting](https://mlsanity.com/paper/2502.06151/) (2025)
- [Towards Closing the Performance Gap for Cryptographic Kernels Between CPUs and Specialized Hardware](https://mlsanity.com/paper/2509.12494/) (2025)

## Built on by (most cited first, 3 of 3)

- [AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems](https://mlsanity.com/paper/2604.03425/) (2026)
- [HE-Guardrail: A Homomorphic Guardrail Against Jailbreak Attacks for Encrypted Large Language Model Inference](https://mlsanity.com/paper/2609.21484/) (2026)
- [Static Bootstrap Placement for Encrypted Language Model Decoding](https://mlsanity.com/paper/2610.04912/) (2026)

## Abstract (arXiv)

Encrypted AI using fully homomorphic encryption (FHE) provides strong privacy guarantees; but its slow performance has limited practical deployment. Recent works proposed ASICs to accelerate FHE, but require expensive advanced manufacturing processes that constrain their accessibility. GPUs are a far more accessible platform, but achieving ASIC-level performance using GPUs has remained elusive. Furthermore, state-of-the-art approaches primarily focus on small models that fit comfortably within a single device. Supporting large models such as LLMs in FHE introduces a dramatic increase in computational complexity that requires optimized GPU kernels, along with managing terabyte-scale memory footprints that far exceed the capacity of a single GPU. This paper presents Cerium, a multi-GPU framework for FHE inference on large models. Cerium integrates a domain-specific language, an optimizing compiler, and a runtime system to automatically generate high-performance GPU kernels, manage terabyte-scale memory footprints, and parallelize computation across multiple GPUs. It introduces new IR constructs, compiler passes, sparse polynomial representations, memory-efficient data layouts, and communication-aware parallelization techniques that together enable encrypted inference for models ranging from small CNNs to Llama3-8B. We build Cerium on NVIDIA GPUs and demonstrate significant performance gains. For small models, Cerium outperforms expert-written hand-optimized GPU libraries by up to 2.25 times. Cerium achieves performance competitive with state-of-the-art FHE ASICs, outright matching prior FHE ASIC CraterLake. It is the first GPU system to execute bootstrapping in under 10 milliseconds, achieving 7.5 milliseconds, and is the first to demonstrate encrypted inference for BERT-Base and Llama3-8B in 8 seconds and 134 seconds, respectively.