# An Attack to Break Permutation-Based Private Third-Party Inference Schemes for LLMs

arXiv: https://arxiv.org/abs/2505.18332 · HTML: https://mlsanity.com/paper/2505.18332/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 13 of 13)

- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence](https://mlsanity.com/paper/2305.03010/) (2023)
- [PUMA: Secure Inference of LLaMA-7B in Five Minutes](https://mlsanity.com/paper/2307.12533/) (2023)
- [Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications](https://mlsanity.com/paper/2408.05148/) (2024)
- [Nimbus: Secure and Efficient Two-Party Inference for Transformers](https://mlsanity.com/paper/2411.15707/) (2024)
- [DAGER: Exact Gradient Inversion for Large Language Models](https://mlsanity.com/paper/2405.15586/) (2024)
- [PermLLM: Private Inference of Large Language Models within 3 Seconds under WAN](https://mlsanity.com/paper/2405.18744/) (2024)
- [InvBERT: Reconstructing Text from Contextualized Word Embeddings by inverting the BERT pipeline](https://mlsanity.com/paper/2109.10104/) (2021)
- [Information Leakage from Embedding in Large Language Models](https://mlsanity.com/paper/2405.11916/) (2024)
- [Secure Transformer Inference Protocol](https://mlsanity.com/paper/2312.00025/) (2023)
- [CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference](https://mlsanity.com/paper/2412.10652/) (2024)
- [Towards Secure and Practical Machine Learning via Secret Sharing and Random Permutation](https://mlsanity.com/paper/2108.07463/) (2021)

## Built on by (most cited first, 2 of 2)

- [Cascade: Token-Sharded Private LLM Inference](https://mlsanity.com/paper/2507.05228/) (2025)
- [Mechanistic Interpretability in the Presence of Architectural Obfuscation](https://mlsanity.com/paper/2506.18053/) (2025)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have led to the widespread adoption of third-party inference services, raising critical privacy concerns. Existing methods of performing private third-party inference, such as Secure Multiparty Computation (SMPC), often rely on cryptographic methods. However, these methods are thousands of times slower than standard unencrypted inference, and fail to scale to large modern LLMs. Therefore, recent lines of work have explored the replacement of expensive encrypted nonlinear computations in SMPC with statistical obfuscation methods - in particular, revealing permuted hidden states to the third parties, with accompanying strong claims of the difficulty of reversal into the unpermuted states. In this work, we begin by introducing a novel reconstruction technique that can recover original prompts from hidden states with nearly perfect accuracy across multiple state-of-the-art LLMs. We then show that extensions of our attack are nearly perfectly effective in reversing permuted hidden states of LLMs, demonstrating the insecurity of three recently proposed privacy schemes. We further dissect the shortcomings of prior theoretical `proofs' of permuation security which allow our attack to succeed. Our findings highlight the importance of rigorous security analysis in privacy-preserving LLM inference.