# A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakage

arXiv: https://arxiv.org/abs/2409.04040 · HTML: https://mlsanity.com/paper/2409.04040/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [A Survey on Homomorphic Encryption Schemes: Theory and Implementation](https://mlsanity.com/paper/1704.03578/) (2017)
- [DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments](https://mlsanity.com/paper/2004.05703/) (2020)
- [TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption](https://mlsanity.com/paper/2104.03152/) (2021)
- [TransLinkGuard: Safeguarding Transformer Models Against Model Stealing in Edge Deployment](https://mlsanity.com/paper/2404.11121/) (2024)
- [LeftoverLocals: Listening to LLM Responses Through Leaked GPU Local Memory](https://mlsanity.com/paper/2401.16603/) (2024)
- [A Survey of Techniques for Improving Security of GPUs](https://mlsanity.com/paper/1804.00114/) (2018)

## Built on by (most cited first, 8 of 8)

- [Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference](https://mlsanity.com/paper/2508.09442/) (2025)
- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations](https://mlsanity.com/paper/2601.10004/) (2026)
- [Watermarking Large Language Models and the Generated Content: Opportunities and Challenges](https://mlsanity.com/paper/2410.19096/) (2024)
- [Lever: Speculative LLM Inference on Smartphones](https://mlsanity.com/paper/2605.16786/) (2026)
- [Good-Enough LLM Obfuscation (GELO)](https://mlsanity.com/paper/2603.05035/) (2026)
- [When Agents Handle Secrets: A Survey of Confidential Computing for Agentic AI](https://mlsanity.com/paper/2605.03213/) (2026)

## Abstract (arXiv)

Running LLMs on end devices has garnered significant attention recently due to their advantages in privacy preservation. With the advent of lightweight LLM models and specially designed GPUs, on-device LLM inference has achieved the necessary accuracy and performance metrics. However, we have identified that LLM inference on GPUs can leak privacy-sensitive intermediate information, specifically the KV pairs. An attacker could exploit these KV pairs to reconstruct the entire user conversation, leading to significant vulnerabilities. Existing solutions, such as Fully Homomorphic Encryption (FHE) and Trusted Execution Environments (TEE), are either too computation-intensive or resource-limited. To address these issues, we designed KV-Shield, which operates in two phases. In the initialization phase, it permutes the weight matrices so that all KV pairs are correspondingly permuted. During the runtime phase, the attention vector is inversely permuted to ensure the correctness of the layer output. All permutation-related operations are executed within the TEE, ensuring that insecure GPUs cannot access the original KV pairs, thus preventing conversation reconstruction. Finally, we theoretically analyze the correctness of KV-Shield, along with its advantages and overhead.