# What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference

arXiv: https://arxiv.org/abs/2605.23158 · HTML: https://mlsanity.com/paper/2605.23158/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 16 of 16)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Explanations from Large Language Models Make Small Reasoners Better](https://mlsanity.com/paper/2210.06726/) (2022)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [Unleashing the Tiger: Inference Attacks on Split Learning](https://mlsanity.com/paper/2012.02670/) (2020)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Bag of Tricks for Training Data Extraction from Language Models](https://mlsanity.com/paper/2302.04460/) (2023)
- [Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications](https://mlsanity.com/paper/2401.07612/) (2024)
- [Gradient Obfuscation Gives a False Sense of Security in Federated Learning](https://mlsanity.com/paper/2206.04055/) (2022)
- [Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation](https://mlsanity.com/paper/2307.04401/) (2023)
- [Language Models are Injective and Hence Invertible](https://mlsanity.com/paper/2510.15511/) (2025)
- [Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack](https://mlsanity.com/paper/2409.00960/) (2024)
- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [Prompt Inference Attack on Distributed Large Language Model Inference Frameworks](https://mlsanity.com/paper/2503.09291/) (2025)
- [Evaluating LLM-based Personal Information Extraction and Countermeasures](https://mlsanity.com/paper/2408.07291/) (2024)

## Built on by (most cited first, 4 of 4)

- [Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines](https://mlsanity.com/paper/2609.34045/) (2026)
- [What Gradients Add to Text Leakage in Split Language Models, Counted per Token and per Document](https://mlsanity.com/paper/2610.04128/) (2026)
- [From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents](https://mlsanity.com/paper/2610.15390/) (2026)
- [A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference](https://mlsanity.com/paper/2610.19607/) (2026)

## Abstract (arXiv)

The deployment of large language models (LLMs) on resource-constrained devices remains challenging, spurring interest in split inference, where models are partitioned between client and server to reduce computational burden and enhance privacy by transmitting only intermediate activations. However, the privacy-preserving capabilities of split inference, particularly in the context of LLMs, have not been exhaustively investigated. To fill this gap, we introduce ActInv, which solves an intermediate activation matching problem to reconstruct the client's input. Extensive evaluations demonstrate that ActInv achieves high-fidelity reconstructions, even in the presence of common perturbation-based defenses such as Gaussian noise injection and activation sparsification. To systematically understand this vulnerability, we develop Perturbation Amplification Factor (PAF), a metric for quantifying a layer's inherent resistance to reconstruction. Our analysis reveals that privacy vulnerability is not uniform across layers, with some layers being highly susceptible to leakage while others offer natural resistance. Furthermore, we demonstrate that defense effectiveness can be significantly improved by calibrating perturbation directions to maximize reconstruction error during backpropagation. Building on these insights, we design PriPert and conduct comprehensive evaluations, covering privacy, utility, and computational overhead, to demonstrate its effectiveness.