mlsanityData to 7 Oct 2026

Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks

arXiv 2602.16760 ↗First posted 18 Feb 2026cs.CR cs.DCcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

We present a practical system for privacy-aware large language model (LLM) inference that splits a transformer between a trusted local GPU and an untrusted cloud GPU, communicating only intermediate activations over the network. Our system addresses the unique challenges of autoregressive LLM decoding over high-latency wide-area networks (WANs), contributing: (1) an asymmetric layer split where embedding and unembedding layers remain local, ensuring raw tokens never leave the trusted device; (2) the first application of lookahead decoding to split inference over WANs, amortizing network round-trip latency across multiple tokens per iteration; (3) an empirical inversion attack evaluation showing that split depth provides a tunable privacy-performance tradeoff -- an attacker can recover ~59%% of tokens at a 2-layer split but only ~35%% at an 8-layer split, with minimal throughput impact; (4) ablation experiments showing that n-gram speculation accepts 1.2-1.3 tokens per decoding step on average (peak of 7 observed on code), with acceptance rates consistent across model scales; (5) formal verification that lookahead decoding produces token-identical output to sequential decoding under greedy argmax, with zero quality degradation; and (6) scaling validation on Mistral NeMo 12B (40 layers), demonstrating that the system generalizes to larger models with only 4.9 GB local VRAM and matching 7B throughput. Evaluated on Mistral 7B and NeMo 12B over a ~80ms WAN link, our system achieves 8.7-9.3 tok/s (7B) and 7.8-8.7 tok/s (12B) with lookahead decoding, with an RTT decomposition model (validated at <6.2%% cross-validation error) projecting 15-19 tok/s at 20ms RTT.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 37 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

23 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges8 Mar

Other references · 22

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  7. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  8. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  9. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  10. 2024CLLMs: Consistency Large Language Modelscited by 71
  11. 2018Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergycited by 50
  12. 2025Language Models are Injective and Hence Invertiblecited by 31
10 more
  1. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 30
  2. 2020DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networkscited by 23
  3. 2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
  4. 2024Adaptive Layer Splitting for Wireless LLM Inference in Edge Computing: A Model-Based Reinforcement Learning Approachcited by 12
  5. 2023PATROL: Privacy-Oriented Pruning for Collaborative Inference Against Model Inversion Attackscited by 9
  6. 2025Model-Distributed Inference for Large Language Models at the Edgecited by 8
  7. 2025Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computingcited by 3
  8. 2025Quantifying Privacy Leakage in Split Inference via Fisher-Approximated Shannon Information Analysiscited by 2
  9. 2025SecureInfer: Heterogeneous TEE-GPU Architecture for Privacy-Critical Tensors for Large Language Model Deploymentcited by 2
  10. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughputcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Confidential and Efficient LLM Inference with Dual Privacy Protection
  2. 2026Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
  3. 2025A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Models
  4. 2026SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs
  5. 2025Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference
  6. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity
  7. 2024When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs
  8. 2026LLMs as Linguistic Chameleons: Decoupling Semantics and Structure for Privacy-Preserving Communication