# PermLLM: Private Inference of Large Language Models within 3 Seconds under WAN

arXiv: https://arxiv.org/abs/2405.18744 · HTML: https://mlsanity.com/paper/2405.18744/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2024.

## Built on by (most cited first, 14 of 14)

- [PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databases](https://mlsanity.com/paper/2506.17336/) (2025)
- [AIDBench: A benchmark for evaluating the authorship identification capability of large language models](https://mlsanity.com/paper/2411.13226/) (2024)
- [SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations](https://mlsanity.com/paper/2601.10004/) (2026)
- [An Attack to Break Permutation-Based Private Third-Party Inference Schemes for LLMs](https://mlsanity.com/paper/2505.18332/) (2025)
- [On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference](https://mlsanity.com/paper/2605.04901/) (2026)
- [Cascade: Token-Sharded Private LLM Inference](https://mlsanity.com/paper/2507.05228/) (2025)
- [P2RAG: Efficient Privacy-Preserving RAG Service Supporting Arbitrary Top-k Retrieval](https://mlsanity.com/paper/2603.14778/) (2026)
- [An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG](https://mlsanity.com/paper/2605.25716/) (2026)
- [Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys](https://mlsanity.com/paper/2609.04382/) (2026)
- [Mechanistic Interpretability in the Presence of Architectural Obfuscation](https://mlsanity.com/paper/2506.18053/) (2025)
- [Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs](https://mlsanity.com/paper/2602.17223/) (2026)
- [Good-Enough LLM Obfuscation (GELO)](https://mlsanity.com/paper/2603.05035/) (2026)
- [When Agents Handle Secrets: A Survey of Confidential Computing for Agentic AI](https://mlsanity.com/paper/2605.03213/) (2026)
- [A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference](https://mlsanity.com/paper/2610.19607/) (2026)

## Abstract (arXiv)

The emergence of ChatGPT marks the arrival of the large language model (LLM) era. While LLMs demonstrate their power in a variety of fields, they also raise serious privacy concerns as the users' queries are sent to the model provider. On the other side, deploying the LLM on the user's device will also leak all the model data. Existing methods based on secure multiparty computation (MPC) managed to protect both the privacy of the model parameters and user queries. However, they require gigabytes of data transfer and several minutes to generate just one token, making them impractical for most real-world applications. To improve the efficiency of private LLM inference, we propose PermLLM, which accelerates the evaluation of non-linear functions using secure random permutation. Along with the optimized secret sharing protocols and homomorphic encryption, PermLLM achieves two-party private inference of the ChatGLM-6B model at the speed of around 3s/token, under a realistic network setting (10ms RTT and 1Gbps bandwidth), which is magnitudes faster than existing MPC solutions.