mlsanityData to 14 Oct 2026

SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding

arXiv 2504.04104 ↗First posted 5 Apr 2025cs.LGcited by 4

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The demand for large language model inference is rapidly increasing. Pipeline parallelism offers a cost-effective deployment strategy for distributed inference but suffers from high service latency. While incorporating speculative decoding to pipeline parallelism improves performance, it still faces challenges of low hardware utilization and narrow speculative window. Inspired by branch prediction in instruction pipelining, we introduce SpecPipe, which fills the pipeline with speculative tokens of a request step-by-step. By maximizing the hardware utilization, SpecPipe decodes one token per pipeline step ideally. Specifically, SpecPipe comprises a dynamic speculative token tree and a pipelined inference framework. The tree dynamically accepts tokens from a speculative token source and outputs the tokens to the inference pipeline. Since the speculative window relaxed in our framework, a high-accuracy draft model is integrated without fine-tuning. The pipeline inference framework follows node-wise computation, pruning propagation, and inter-node communication stages. We implement SpecPipe and a variant SpecPipe-DB with dynamic batching for single- and multi-request inference, respectively. On an 8-stage pipeline, SpecPipe improves time between tokens on diverse single-request workloads by 4.19×-5.53× over standard pipeline parallelism and by 2.08×-2.38× over prior tree-based speculative decoding methods. For multi-request workloads, SpecPipe-DB achieves 1.64×-2.08× higher throughput and 1.61×-2.06× lower time between tokens than vLLM.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 56 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

31 of 56 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoning31 Jan
  2. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  3. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation16 Jul
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  8. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  9. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  11. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  10. 2020Aligning AI With Shared Human Valuescited by 723
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
8 more
  1. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  3. 2020FALCON: Honest-Majority Maliciously Secure Framework for Private Deep Learningcited by 74
  4. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  5. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  6. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  7. 2023Large language models in medicine: the potentials and pitfallscited by 21
  8. 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  2. 2026SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agentscited by 2
  3. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  4. 2026Speculative Pipeline Decoding: Higher-Accuracy Drafting with Hidden Latency via Pipeline Parallelismcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  2. 2025PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
  3. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
  4. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  5. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  6. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  7. 2025SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism
  8. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput