# THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

arXiv: https://arxiv.org/abs/2206.00216 · HTML: https://mlsanity.com/paper/2206.00216/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1904.02232/) (2019)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [SOFT: Softmax-free Transformer with Linear Complexity](https://mlsanity.com/paper/2110.11945/) (2021)
- [Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness](https://mlsanity.com/paper/2010.01285/) (2020)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Generalised Differential Privacy for Text Document Processing](https://mlsanity.com/paper/1811.10256/) (2018)
- [Differentially Private Language Models Benefit from Public Pre-training](https://mlsanity.com/paper/2009.05886/) (2020)
- [Simple Applications of BERT for Ad Hoc Document Retrieval](https://mlsanity.com/paper/1903.10972/) (2019)
- [TextHide: Tackling Data Privacy in Language Understanding Tasks](https://mlsanity.com/paper/2010.06053/) (2020)
- [Benchmarking Differential Privacy and Federated Learning for BERT Models](https://mlsanity.com/paper/2106.13973/) (2021)
- [nGraph-HE: A Graph Compiler for Deep Learning on Homomorphically Encrypted Data](https://mlsanity.com/paper/1810.10121/) (2018)
- [nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data](https://mlsanity.com/paper/1908.04172/) (2019)
- [Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning](https://mlsanity.com/paper/2110.01643/) (2021)
- [Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets](https://mlsanity.com/paper/1412.7091/) (2014)

## Built on by (most cited first, 30 of 57)

- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Privacy in Large Language Models: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2310.10383/) (2023)
- [PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models](https://mlsanity.com/paper/2311.04044/) (2023)
- [Split-and-Denoise: Protect large language model inference with local differential privacy](https://mlsanity.com/paper/2310.09130/) (2023)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Grounding Foundation Models through Federated Transfer Learning: A General Framework](https://mlsanity.com/paper/2311.17431/) (2023)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers](https://mlsanity.com/paper/2305.18396/) (2023)
- [A Survey on Privacy Risks and Protection in Large Language Models](https://mlsanity.com/paper/2505.01976/) (2025)
- [MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention](https://mlsanity.com/paper/2211.13955/) (2022)
- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [Nimbus: Secure and Efficient Two-Party Inference for Transformers](https://mlsanity.com/paper/2411.15707/) (2024)
- [Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives](https://mlsanity.com/paper/2411.05818/) (2024)
- [CipherPrune: Efficient and Scalable Private Transformer Inference](https://mlsanity.com/paper/2502.16782/) (2025)
- [MERGE: Fast Private Text Generation](https://mlsanity.com/paper/2305.15769/) (2023)
- [East: Efficient and Accurate Secure Transformer Framework for Inference](https://mlsanity.com/paper/2308.09923/) (2023)
- [SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC](https://mlsanity.com/paper/2401.00793/) (2024)
- [Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks](https://mlsanity.com/paper/2310.13291/) (2023)
- [Privacy-Preserving Language Model Inference with Instance Obfuscation](https://mlsanity.com/paper/2402.08227/) (2024)
- [Power-Softmax: Towards Secure LLM Inference over Encrypted Data](https://mlsanity.com/paper/2410.09457/) (2024)
- [Efficiency 360: Efficient Vision Transformers](https://mlsanity.com/paper/2302.08374/) (2023)
- [Secure Transformer Inference Protocol](https://mlsanity.com/paper/2312.00025/) (2023)
- [MPC-Minimized Secure LLM Inference](https://mlsanity.com/paper/2408.03561/) (2024)
- [CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference](https://mlsanity.com/paper/2412.10652/) (2024)
- [LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliance](https://mlsanity.com/paper/2505.17145/) (2025)
- [CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model](https://mlsanity.com/paper/2310.15477/) (2023)
- [Privacy-Preserving Diffusion Model Using Homomorphic Encryption](https://mlsanity.com/paper/2403.05794/) (2024)
- [Preserving Privacy and Utility in LLM-Based Product Recommendations](https://mlsanity.com/paper/2505.00951/) (2025)

## Abstract (arXiv)

As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medical record, bank account). Privacy-preserving inference of transformer models is on the demand of cloud service users. To protect privacy, it is an attractive choice to compute only with ciphertext in homomorphic encryption (HE). However, enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce textitTHE-X, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. textitTHE-X proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and LayerNorm. Experiments reveal our proposed textitTHE-X can enable transformer inference on encrypted data for different downstream tasks, all with negligible performance drop but enjoying the theory-guaranteed privacy-preserving advantage.