mlsanityData to 7 Oct 2026

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

arXiv 2206.00216 ↗First posted 1 Jun 2022cs.CR cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medical record, bank account). Privacy-preserving inference of transformer models is on the demand of cloud service users. To protect privacy, it is an attractive choice to compute only with ciphertext in homomorphic encryption (HE). However, enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce textitTHE-X, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. textitTHE-X proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and LayerNorm. Experiments reveal our proposed textitTHE-X can enable transformer inference on encrypted data for different downstream tasks, all with negligible performance drop but enjoying the theory-guaranteed privacy-preserving advantage.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

26 of 47 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2016Layer Normalizationcited by 5,579
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019Deep Leakage from Gradientscited by 1,004
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2020iDLG: Improved Deep Leakage from Gradientscited by 367
  12. 2019Understanding and Improving Layer Normalizationcited by 213
14 more
  1. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  2. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  3. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  4. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 66
  5. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 54
  6. 2018Generalised Differential Privacy for Text Document Processingcited by 52
  7. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 46
  8. 2019Simple Applications of BERT for Ad Hoc Document Retrievalcited by 34
  9. 2020TextHide: Tackling Data Privacy in Language Understanding Taskscited by 32
  10. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28
  11. 2018nGraph-HE: A Graph Compiler for Deep Learning on Homomorphically Encrypted Datacited by 24
  12. 2019nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Datacited by 21
  13. 2021Privacy enabled Financial Text Classification using Differential Privacy and Federated Learningcited by 18
  14. 2014Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targetscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 16; 2025: 12; 2026: 10.

55 papers.

Sort:

By area:

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  3. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 35
  4. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 30
  5. 2023A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacycited by 24
  6. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  7. 2024Privacy Preserving Prompt Engineering: A Surveycited by 24
  8. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  9. 2023LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformerscited by 20
  10. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  11. 2023Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryptioncited by 19
  12. 2024Encryption-Friendly LLM Architecturecited by 15
38 more
  1. 2024Nimbus: Secure and Efficient Two-Party Inference for Transformerscited by 15
  2. 2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
  3. 2023East: Efficient and Accurate Secure Transformer Framework for Inferencecited by 9
  4. 2025CipherPrune: Efficient and Scalable Private Transformer Inferencecited by 9
  5. 2023MERGE: Fast Private Text Generationcited by 8
  6. 2023Assessing Privacy Risks in Language Models: A Case Study on Summarization Taskscited by 8
  7. 2024SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPCcited by 7
  8. 2023Efficiency 360: Efficient Vision Transformerscited by 6
  9. 2024Privacy-Preserving Language Model Inference with Instance Obfuscationcited by 6
  10. 2024Power-Softmax: Towards Secure LLM Inference over Encrypted Datacited by 6
  11. 2024CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inferencecited by 6
  12. 2025LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliancecited by 6
  13. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  14. 2023Secure Transformer Inference Protocolcited by 5
  15. 2024MPC-Minimized Secure LLM Inferencecited by 5
  16. 2025Preserving Privacy and Utility in LLM-Based Product Recommendationscited by 5
  17. 2024Privacy-Preserving Diffusion Model Using Homomorphic Encryptioncited by 4
  18. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  19. 2023ConfusionPrompt: Practical Private Inference for Online Large Language Modelscited by 3
  20. 2024A Survey on Privacy Attacks Against Digital Twin Systems in AI-Roboticscited by 3
  21. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 3
  22. 2025MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstractioncited by 3
  23. 2023LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloudcited by 2
  24. 2023SelectFormer: Private and Practical Data Selection for Transformerscited by 1
  25. 2024A Framework for Cost-Effective and Self-Adaptive LLM Shaking and Recovery Mechanismcited by 1
  26. 2024Confidential Prompting: Privacy-preserving LLM Inference on Cloudcited by 1
  27. 2025Private Transformer Inference in MLaaS: A Surveycited by 1
  28. 2025ENSI: Efficient Non-Interactive Secure Inference for Large Language Modelscited by 1
  29. 2026CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryptioncited by 1
  30. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  31. 2026On the (In-)Security of the Shuffling Defense in the Transformer Secure Inferencecited by 1
  32. 2024Optimizing Privacy and Utility Tradeoffs for Group Interests Through Harmonizationcited by 0
  33. 2025Stereo-LiDAR Fusion by Semi-Global Matching With Discrete Disparity-Matching Cost and Semidensificationcited by 0
  34. 2025An Efficient Private GPT Never Autoregressively Decodescited by 0
  35. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMscited by 0
  36. 2025FHE-Agent: Automating CKKS Configuration for Practical Encrypted Inference via an LLM-Guided Agentic Frameworkcited by 0
  37. 2026mmFHE: mmWave Sensing with End-to-End Fully Homomorphic Encryptioncited by 0
  38. 2026Public-Decay Homomorphic State Space Models for Private Sequence Inferencecited by 0

50 most cited of 55 All 55 (JSON)