# Prompt Inference Attack on Distributed Large Language Model Inference Frameworks

arXiv: https://arxiv.org/abs/2503.09291 · HTML: https://mlsanity.com/paper/2503.09291/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2025.

## Built on by (most cited first, 14 of 14)

- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference](https://mlsanity.com/paper/2605.23158/) (2026)
- [DREAM: Dynamic Red-teaming across Environments for AI Models](https://mlsanity.com/paper/2512.19016/) (2025)
- [Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models](https://mlsanity.com/paper/2603.00196/) (2026)
- [Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys](https://mlsanity.com/paper/2609.04382/) (2026)
- [A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendation](https://mlsanity.com/paper/2506.11563/) (2025)
- [Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions](https://mlsanity.com/paper/2510.22909/) (2025)
- [Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models](https://mlsanity.com/paper/2607.02819/) (2026)
- [Securing LLMs in the Wild: Privacy and Security Challenges at the Edge](https://mlsanity.com/paper/2607.13088/) (2026)
- [Integrity of peer-to-peer distributed LLM inference under malicious nodes](https://mlsanity.com/paper/2607.19490/) (2026)
- [Benchmarking Text-to-SQL under Role-Based Access Control](https://mlsanity.com/paper/2607.22115/) (2026)
- [Denoising-Aware Inversion: Revealing Privacy Risks in Noise-Protected Text Embeddings](https://mlsanity.com/paper/2608.18610/) (2026)
- [Auditing and Mitigating Privacy Leakage in Cloud-Edge Collaborative Decoding](https://mlsanity.com/paper/2608.29111/) (2026)
- [SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs](https://mlsanity.com/paper/2609.38830/) (2026)

## Abstract (arXiv)

The inference process of modern large language models (LLMs) demands prohibitive computational resources, rendering them infeasible for deployment on consumer-grade devices. To address this limitation, recent studies propose distributed LLM inference frameworks, which employ split learning principles to enable collaborative LLM inference on resource-constrained hardware. However, distributing LLM layers across participants requires the transmission of intermediate outputs, which may introduce privacy risks to the original input prompts - a critical issue that has yet to be thoroughly explored in the literature. In this paper, we rigorously examine the privacy vulnerabilities of distributed LLM inference frameworks by designing and evaluating three prompt inference attacks aimed at reconstructing input prompts from intermediate LLM outputs. These attacks are developed under various query and data constraints to reflect diverse real-world LLM service scenarios. Specifically, the first attack assumes an unlimited query budget and access to an auxiliary dataset sharing the same distribution as the target prompts. The second attack also leverages unlimited queries but uses an auxiliary dataset with a distribution differing from the target prompts. The third attack operates under the most restrictive scenario, with limited query budgets and no auxiliary dataset available. We evaluate these attacks on a range of LLMs, including state-of-the-art models such as Llama-3.2 and Phi-3.5, as well as widely-used models like GPT-2 and BERT for comparative analysis. Our experiments show that the first two attacks achieve reconstruction accuracies exceeding 90%, while the third achieves accuracies typically above 50%, even under stringent constraints. These findings highlight privacy risks in distributed LLM inference frameworks, issuing a strong alert on their deployment in real-world applications.