# Towards Confidential and Efficient LLM Inference with Dual Privacy Protection

arXiv: https://arxiv.org/abs/2509.09091 · HTML: https://mlsanity.com/paper/2509.09091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models](https://mlsanity.com/paper/2309.11674/) (2023)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass](https://mlsanity.com/paper/2309.06746/) (2023)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Local, Private, Efficient Protocols for Succinct Histograms](https://mlsanity.com/paper/1504.04686/) (2015)
- [ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks](https://mlsanity.com/paper/2011.05905/) (2020)
- [No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML](https://mlsanity.com/paper/2310.07152/) (2023)
- [DarKnight: An Accelerated Framework for Privacy and Integrity Preserving Deep Learning Using Trusted Hardware](https://mlsanity.com/paper/2207.00083/) (2022)
- [Occlum: Secure and Efficient Multitasking Inside a Single Enclave of Intel SGX](https://mlsanity.com/paper/2001.07450/) (2020)
- [Confidential Machine Learning Computation in Untrusted Environments: A Systems Security Perspective](https://mlsanity.com/paper/2111.03308/) (2021)
- [Tempo: Confidentiality Preservation in Cloud-Based Neural Network Training](https://mlsanity.com/paper/2401.11531/) (2024)

## Built on by (most cited first, 1 of 1)

- [When Agents Handle Secrets: A Survey of Confidential Computing for Agentic AI](https://mlsanity.com/paper/2605.03213/) (2026)

## Abstract (arXiv)

CPU-based trusted execution environments (TEEs) and differential privacy (DP) have gained wide applications for private inference. Due to high inference latency in TEEs, researchers use partition-based approaches that offload linear model components to GPUs. However, dense nonlinear layers of large language models (LLMs) result in significant communication overhead between TEEs and GPUs. DP-based approaches apply random noise to protect data privacy, but this compromises LLM performance and semantic understanding. To overcome the above drawbacks, this paper proposes CMIF, a Confidential and efficient Model Inference Framework. CMIF confidentially deploys the embedding layer in the client-side TEE and subsequent layers on GPU servers. Meanwhile, it optimizes the Report-Noisy-Max mechanism to protect sensitive inputs with a slight decrease in model performance. Extensive experiments on Llama-series models demonstrate that CMIF reduces additional inference overhead in TEEs while preserving user data privacy.