Data to 7 Oct 2026

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

arXiv 2607.13093 ↗First posted 14 Jul 2026cs.CR cs.AIcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1% and downlink payloads by up to 67.4% over baseline split inference, retaining comparable performance to full cloud inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 51 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

33 of 51 references.

Closest ancestors · 3

Same direction, newest first

Other references · 30

2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776

18 more

2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 79
2022MPCFormer: fast, performant and private Transformer inference with MPCcited by 67
2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 65
2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 55
2023Privacy Side Channels in Machine Learning Systemscited by 38
2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 30
2022Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformerscited by 28
2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
2023Efficient LLM Inference on CPUscited by 24
2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challengescited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way