mlsanityData to 14 Oct 2026

Encryption-Friendly LLM Architecture

arXiv 2410.02486 ↗First posted 3 Oct 2024cs.CR cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups -- 6.94x for fine-tuning and 2.3x for inference -- while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial. Our code is available on GitHub.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 40 of its 86 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

40 of 86 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
28 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  11. 2018Marian: Fast Neural Machine Translation in C++cited by 203
  12. 2024Stealing Part of a Production Language Modelcited by 159
  13. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  14. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  15. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  16. 2022MPCFormer: fast, performant and private Transformer inference with MPCcited by 69
  17. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
  18. 2021Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Networkcited by 57
  19. 2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 57
  20. 2023PUMA: Secure Inference of LLaMA-7B in Five Minutescited by 48
  21. 2021Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Datacited by 27
  22. 2020Normalized Attention Without Probability Cagecited by 20
  23. 2023Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryptioncited by 19
  24. 2019PrivFT: Private and Fast Text Classification with Homomorphic Encryptioncited by 12
  25. 2024HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryptioncited by 12
  26. 2022Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryptioncited by 9
  27. 2023Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryptioncited by 5
  28. 2024PrivTuner with Homomorphic Encryption and LoRA: A P3EFT Scheme for Privacy-Preserving Parameter-Efficient Fine-Tuning of AI Foundation Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  3. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  4. 2025A Scalable Multi-GPU Framework for Encrypted Large-Model Inferencecited by 3
  5. 2025ENSI: Efficient Non-Interactive Secure Inference for Large Language Modelscited by 2
  6. 2026ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hourcited by 2
  7. 2025SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPCcited by 1
  8. 2025Traveling Salesman-Based Token Ordering Improves Stability in Homomorphically Encrypted Language Modelscited by 1
  9. 2026CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryptioncited by 1
  10. 2026Federated Inference: Toward Privacy-Preserving Collaborative and Incentivized Model Servingcited by 1
  11. 2026The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Modelscited by 1
  12. 2025CryptoTensors: A Light-Weight Large Language Model File Format for Highly-Secure Model Distributioncited by 0
5 more
  1. 2026Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Modelscited by 0
  2. 2026HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptationcited by 0
  3. 2026HE-Guardrail: A Homomorphic Guardrail Against Jailbreak Attacks for Encrypted Large Language Model Inferencecited by 0
  4. 2026Static Bootstrap Placement for Encrypted Language Model Decodingcited by 0
  5. 2026A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inferencecited by 0