# Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption

arXiv: https://arxiv.org/abs/2311.08610 · HTML: https://mlsanity.com/paper/2311.08610/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network](https://mlsanity.com/paper/2106.07229/) (2021)
- [THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption](https://mlsanity.com/paper/2206.00216/) (2022)
- [CryptoDL: Deep Neural Networks over Encrypted Data](https://mlsanity.com/paper/1711.05189/) (2017)
- [Pi-nets: Deep Polynomial Neural Networks](https://mlsanity.com/paper/2003.03828/) (2020)
- [Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data](https://mlsanity.com/paper/2105.10879/) (2021)
- [HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data](https://mlsanity.com/paper/2011.01805/) (2020)
- [LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers](https://mlsanity.com/paper/2305.18396/) (2023)
- [MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention](https://mlsanity.com/paper/2211.13955/) (2022)
- [Primer: Fast Private Transformer Inference on Encrypted Data](https://mlsanity.com/paper/2303.13679/) (2023)
- [AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE](https://mlsanity.com/paper/2310.08012/) (2023)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)
- [A methodology for training homomorphicencryption friendly neural networks](https://mlsanity.com/paper/2111.03362/) (2021)
- [Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption](https://mlsanity.com/paper/2210.02574/) (2022)
- [Privacy preserving Neural Network Inference on Encrypted Data with GPUs](https://mlsanity.com/paper/1911.11377/) (2019)

## Built on by (most cited first, 19 of 19)

- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [SLIP: Securing LLMs IP Using Weights Decomposition](https://mlsanity.com/paper/2407.10886/) (2024)
- [CipherPrune: Efficient and Scalable Private Transformer Inference](https://mlsanity.com/paper/2502.16782/) (2025)
- [Power-Softmax: Towards Secure LLM Inference over Encrypted Data](https://mlsanity.com/paper/2410.09457/) (2024)
- [LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliance](https://mlsanity.com/paper/2505.17145/) (2025)
- [Efficient Decoding Methods for Language Models on Encrypted Data](https://mlsanity.com/paper/2509.08383/) (2025)
- [On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach](https://mlsanity.com/paper/2502.02209/) (2025)
- [ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models](https://mlsanity.com/paper/2410.09637/) (2024)
- [AERO: Entropy-Guided Framework for Private LLM Inference](https://mlsanity.com/paper/2410.13060/) (2024)
- [A Training Framework for Optimal and Stable Training of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11589/) (2025)
- [An Efficient Private GPT Never Autoregressively Decodes](https://mlsanity.com/paper/2505.15252/) (2025)
- [ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour](https://mlsanity.com/paper/2607.23478/) (2026)
- [FuseFSS: Efficient Secure LLM Inference with Function Secret Sharing](https://mlsanity.com/paper/2606.09551/) (2026)
- [The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Models](https://mlsanity.com/paper/2606.16801/) (2026)
- [Efficient Privacy-Preserving KAN Inference Using Homomorphic Encryption](https://mlsanity.com/paper/2409.07751/) (2024)
- [Gecko: Fast Private Inference via Secure Public Encoder Offloading](https://mlsanity.com/paper/2608.02378/) (2026)
- [HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation](https://mlsanity.com/paper/2609.01730/) (2026)
- [DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference](https://mlsanity.com/paper/2609.32204/) (2026)

## Abstract (arXiv)

Designing privacy-preserving deep learning models is a major challenge within the deep learning community. Homomorphic Encryption (HE) has emerged as one of the most promising approaches in this realm, enabling the decoupling of knowledge between the model owner and the data owner. Despite extensive research and application of this technology, primarily in convolutional neural networks, incorporating HE into transformer models has been challenging because of the difficulties in converting these models into a polynomial form. We break new ground by introducing the first polynomial transformer, providing the first demonstration of secure inference over HE with transformers. This includes a transformer architecture tailored for HE, alongside a novel method for converting operators to their polynomial equivalent. This innovation enables us to perform secure inference on LMs with WikiText-103. It also allows us to perform image classification with CIFAR-100 and Tiny-ImageNet. Our models yield results comparable to traditional methods, bridging the performance gap with transformers of similar scale and underscoring the viability of HE for state-of-the-art applications. Finally, we assess the stability of our models and conduct a series of ablations to quantify the contribution of each model component.