# Auditing and Mitigating Privacy Leakage in Cloud-Edge Collaborative Decoding

arXiv: https://arxiv.org/abs/2608.29111 · HTML: https://mlsanity.com/paper/2608.29111/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 29 Aug 2026.

## Builds on (most cited first, 23 of 23)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting](https://mlsanity.com/paper/2501.13987/) (2025)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Extracting Prompts by Inverting LLM Outputs](https://mlsanity.com/paper/2405.15012/) (2024)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following](https://mlsanity.com/paper/2403.03129/) (2024)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing](https://mlsanity.com/paper/2502.01976/) (2025)
- [Prompt Inference Attack on Distributed Large Language Model Inference Frameworks](https://mlsanity.com/paper/2503.09291/) (2025)
- [AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment](https://mlsanity.com/paper/2411.10606/) (2024)
- [InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy](https://mlsanity.com/paper/2507.02974/) (2025)
- [PATROL: Privacy-Oriented Pruning for Collaborative Inference Against Model Inversion Attacks](https://mlsanity.com/paper/2307.10981/) (2023)
- [Better Language Model Inversion by Compactly Representing Next-Token Distributions](https://mlsanity.com/paper/2506.17090/) (2025)
- [Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models](https://mlsanity.com/paper/2410.03026/) (2024)
- [SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot](https://mlsanity.com/paper/2506.09613/) (2025)
- [GPT, But Backwards: Exactly Inverting Language Model Outputs](https://mlsanity.com/paper/2507.01693/) (2025)
- [Locally Optimal Private Sampling: Beyond the Global Minimax](https://mlsanity.com/paper/2510.09485/) (2025)
- [Floe: Federated Specialization for Real-Time LLM-SLM Inference](https://mlsanity.com/paper/2602.14302/) (2026)
- [dMoE: dLLMs with Learnable Block Experts](https://mlsanity.com/paper/2605.30876/) (2026)

## Abstract (arXiv)

Applications such as personalized assistance and proprietary document analysis require large language models (LLMs) to generate outputs from private data. Yet powerful LLMs typically cannot be deployed on the resource-constrained devices where private data resides, and uploading private data to cloud-hosted LLMs exposes sensitive information. Recent work addresses this tension with a cloud-edge collaborative decoding paradigm, where private data are kept on the edge with a small language model (SLM) producing next-token distributions, which are fused with predictions from a cloud LLM operating solely on public data. In this paper, we systematically analyze the privacy risks of such a paradigm with a novel evaluation framework using constructed QA datasets, which show that such collaboration can expose substantial private-context information. To address such privacy leakage, we propose CoVeil, a defense mechanism which dynamically optimizes transmitted signals to suppress leakage during decoding time while preserving the collaborative quality. Extensive evaluations demonstrate that CoVeil consistently improves the privacy-utility trade-off over existing baselines by reducing data leakage by up to 87.2%, with minimal accuracy loss.