# From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models

arXiv: https://arxiv.org/abs/2606.14210 · HTML: https://mlsanity.com/paper/2606.14210/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Label Leakage and Protection in Two-party Split Learning](https://mlsanity.com/paper/2102.08504/) (2021)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass](https://mlsanity.com/paper/2309.06746/) (2023)
- [TAG: Gradient Attack on Transformer-based Language Models](https://mlsanity.com/paper/2103.06819/) (2021)
- [SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2407.00952/) (2024)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Split-and-Denoise: Protect large language model inference with local differential privacy](https://mlsanity.com/paper/2310.09130/) (2023)
- [Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack](https://mlsanity.com/paper/2409.00960/) (2024)
- [Prompt Inversion Attack against Collaborative Inference of Large Language Models](https://mlsanity.com/paper/2503.09022/) (2025)
- [A Split-and-Privatize Framework for Large Language Model Fine-Tuning](https://mlsanity.com/paper/2312.15603/) (2023)
- [VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMs](https://mlsanity.com/paper/2508.03097/) (2025)
- [PrivaScissors: Enhance the Privacy of Collaborative Inference through the Lens of Mutual Information](https://mlsanity.com/paper/2306.07973/) (2023)

## Built on by (most cited first, 2 of 2)

- [Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys](https://mlsanity.com/paper/2609.04382/) (2026)
- [What Gradients Add to Text Leakage in Split Language Models, Counted per Token and per Document](https://mlsanity.com/paper/2610.04128/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in privacy-sensitive domains, where users must balance the risk of data exposure through external APIs against the high computational cost of local deployment. Split learning has therefore emerged as a promising paradigm for LLM fine-tuning and inference under limited local resources. However, it introduces new privacy risks. Prior work primarily studies leakage of private input prompts, typically via inversion attacks on intermediate representations, while the potential for sensitive information leakage through generative response outputs remains largely unexplored. In this work, we unveil novel vulnerabilities of Split-LLM by presenting Patched Model Inversion with Dual-Sided Initialization (PIDI), a two-stage attack that simultaneously targets both private input prompts and output responses in Split-LLM settings. It combines dual-sided initialization with a patched inversion strategy to tackle long sequences, substantially outperforming prior inversion methods. To counter threats from both sides, we further propose the Adapter-based DualGuard with Mutual Information Defense (ADMI), which integrates an adapter-based local warmup strategy and mutual information regularization to provide a strong empirical privacy protection with minimal impact on task performance. Extensive experiments across diverse tasks and models demonstrate that ADMI effectively defends against PIDI and other state-of-the-art inversion attacks. Our code is publicly available at https://github.com/FLAIR-THU/VFLAIR-LLM.