# Leveraging ASIC AI Chips for Homomorphic Encryption

arXiv: https://arxiv.org/abs/2501.07047 · HTML: https://mlsanity.com/paper/2501.07047/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 94 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Low Latency Privacy Preserving Inference](https://mlsanity.com/paper/1812.10659/) (2018)
- [Generative Artificial Intelligence: A Systematic Review and Applications](https://mlsanity.com/paper/2405.11029/) (2024)
- [F1: A Fast and Programmable Accelerator for Fully Homomorphic Encryption (Extended Version)](https://mlsanity.com/paper/2109.05371/) (2021)
- [Orion: A Fully Homomorphic Encryption Framework for Deep Learning](https://mlsanity.com/paper/2311.03470/) (2023)
- [BTS: An Accelerator for Bootstrappable Fully Homomorphic Encryption](https://mlsanity.com/paper/2112.15479/) (2021)
- [ARK: Fully Homomorphic Encryption Accelerator with Runtime Data Generation and Inter-Operation Key Reuse](https://mlsanity.com/paper/2205.00922/) (2022)
- [HEAX: An Architecture for Computing on Encrypted Data](https://mlsanity.com/paper/1909.09731/) (2019)
- [TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU](https://mlsanity.com/paper/2212.14191/) (2022)
- [GME: GPU-based Microarchitectural Extensions to Accelerate Homomorphic Encryption](https://mlsanity.com/paper/2309.11001/) (2023)
- [FAB: An FPGA-based Accelerator for Bootstrappable Fully Homomorphic Encryption](https://mlsanity.com/paper/2207.11872/) (2022)
- [Cheddar: A Swift Fully Homomorphic Encryption Library Designed for GPU Architectures](https://mlsanity.com/paper/2407.13055/) (2024)
- [Accelerating Encrypted Computing on Intel GPUs](https://mlsanity.com/paper/2109.14704/) (2021)
- [Accelerating Polynomial Multiplication for Homomorphic Encryption on GPUs](https://mlsanity.com/paper/2209.01290/) (2022)
- [SoK: Fully Homomorphic Encryption Accelerators](https://mlsanity.com/paper/2212.01713/) (2022)
- [CHET: Compiler and Runtime for Homomorphic Evaluation of Tensor Programs](https://mlsanity.com/paper/1810.00845/) (2018)
- [Accelerating Number Theoretic Transformations for Bootstrappable Homomorphic Encryption on GPUs](https://mlsanity.com/paper/2012.01968/) (2020)
- [Does Fully Homomorphic Encryption Need Compute Acceleration?](https://mlsanity.com/paper/2112.06396/) (2021)
- [BASALISC: Programmable Hardware Accelerator for BGV Fully Homomorphic Encryption](https://mlsanity.com/paper/2205.14017/) (2022)
- [Strix: An End-to-End Streaming Architecture with Two-Level Ciphertext Batching for Fully Homomorphic Encryption with Programmable Bootstrapping](https://mlsanity.com/paper/2305.11423/) (2023)
- [TPU as Cryptographic Accelerator](https://mlsanity.com/paper/2307.06554/) (2023)
- [Chameleon: An Efficient FHE Scheme Switching Acceleration on GPUs](https://mlsanity.com/paper/2410.05934/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Scalable Multi-GPU Framework for Encrypted Large-Model Inference](https://mlsanity.com/paper/2512.11269/) (2025)
- [Privatar: Scalable Privacy-preserving Multi-user VR via Secure Offloading](https://mlsanity.com/paper/2604.17476/) (2026)
- [Enabling AI ASICs for Zero Knowledge Proof](https://mlsanity.com/paper/2604.17808/) (2026)

## Abstract (arXiv)

Homomorphic Encryption (HE) provides strong data privacy for cloud services but at the cost of prohibitive computational overhead. While GPUs have emerged as a practical platform for accelerating HE, there remains an order-of-magnitude energy-efficiency gap compared to specialized (but expensive) HE ASICs. This paper explores an alternate direction: leveraging existing AI accelerators, like Google's TPUs with coarse-grained compute and memory architectures, to offer a path toward ASIC-level energy efficiency for HE. However, this architectural paradigm creates a fundamental mismatch with SoTA HE algorithms designed for GPUs. These algorithms rely heavily on: (1) high-precision (32-bit) integer arithmetic to now run on a TPU's low-throughput vector unit, leaving its high-throughput low-precision (8-bit) matrix engine (MXU) idle, and (2) fine-grained data permutations that are inefficient on the TPU's coarse-grained memory subsystem. Consequently, porting GPU-optimized HE libraries to TPUs results in severe resource under-utilization and performance degradation. To tackle above challenges, we introduce CROSS, a compiler framework that systematically transforms HE workloads to align with the TPU's architecture. CROSS makes two key contributions: (1) Basis-Aligned Transformation (BAT), a novel technique that converts high-precision modular arithmetic into dense, low-precision (INT8) matrix multiplications, unlocking and improving the utilization of TPU's MXU for HE, and (2) Memory-Aligned Transformation (MAT), which eliminates costly runtime data reordering by embedding reordering into compute kernels through offline parameter transformation. CROSS (TPU v6e) achieves higher throughput per watt on NTT and HE operators than WarpDrive, FIDESlib, FAB, HEAP, and Cheddar, establishing AI ASIC as the SotA efficient platform for HE operators. Code: https://github.com/EfficientPPML/CROSS