# PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption

arXiv: https://arxiv.org/abs/2411.03357 · HTML: https://mlsanity.com/paper/2411.03357/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2024.

## Built on by (most cited first, 10 of 10)

- [Trusted AI Agents in the Cloud](https://mlsanity.com/paper/2512.05951/) (2025)
- [IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation](https://mlsanity.com/paper/2602.22700/) (2026)
- [Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI](https://mlsanity.com/paper/2603.07466/) (2026)
- [RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU](https://mlsanity.com/paper/2504.15302/) (2025)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)
- [Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)](https://mlsanity.com/paper/2603.01499/) (2026)
- [The Proxy Knows Too Much: Sealing LLM API Routers with Attested TEEs](https://mlsanity.com/paper/2606.16358/) (2026)
- [Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environments](https://mlsanity.com/paper/2602.19450/) (2026)
- [The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing](https://mlsanity.com/paper/2606.23969/) (2026)
- [SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels](https://mlsanity.com/paper/2608.02995/) (2026)

## Abstract (arXiv)

Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language Models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8 percent and 88.2 percent throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining - an idea inspired by the CPU instruction pipelining - thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments on NVIDIA H100 GPU show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (less than 19.6 percent in throughput) across various LLM sizes, from 13B to 175B.