mlsanityData to 7 Oct 2026

PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding

arXiv 2605.13319 ↗First posted 13 May 2026cs.DCcited by 4

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Speculative decoding can significantly accelerate LLM inference, especially given that its cloud-edge collaborative deployment offers cloud workload offloading, offline robustness, and privacy enhancement. However, existing collaborative inference frameworks with speculative decoding are constrained by (i) sequential token generation and communication with low resource utilization, and (ii) inflexible cloud non-autoregressive verification (NAV) triggering that induces premature verification or costly rollbacks. In this paper, we propose PipeSD, an efficient cloud-edge collaborative pipeline inference framework with speculative decoding. PipeSD overlaps token generation and communication by a token-batch pipeline scheduling mechanism optimized by dynamic programming, and improves verification flexibility through a dual-threshold NAV triggering mechanism with a lightweight Bayesian optimization autotuner. We implement PipeSD using llama-cpp-python, PyTorch, and FastAPI, and evaluate it on a real-world cloud-edge testbed with two draft-target model pairs across four scenarios. Results show that PipeSD consistently outperforms state-of-the-art baselines, achieving 1.16x-2.16x speedup and reducing energy consumption by 14.3%-25.3%.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 44 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

25 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Models23 Mar
  2. 2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks16 Jan

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
11 more
  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  2. 2023Speculative Decoding with Big Little Decodercited by 163
  3. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  4. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  5. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  6. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 15
  7. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challengescited by 14
  8. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  9. 2024An Edge-Cloud Collaboration Framework for Generative AI Service Provision with Synergetic Big Cloud Model and Small Edge Modelscited by 9
  10. 2023EcoFed: Efficient Communication for DNN Partitioning-based Federated Learningcited by 5
  11. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Trainingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Offcited by 1
  2. 2026AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networkscited by 0
  3. 2026AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inferencecited by 0
  4. 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
  2. 2026FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decoding
  3. 2026ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
  4. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
  5. 2025DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving
  6. 2026Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
  7. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  8. 2025PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding