Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding
On the map: Efficient LLM inference and serving → Speculative decoding
Abstract (arXiv)
Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost because each output token is generated auto-regressively through all model layers. Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost. However, in practice, many approaches struggle to achieve the expected acceleration in such draft-then-verify paradigm even with a well-aligned early-exit head and selected exit position. Our analysis reveals that EESD only pays off when the vast majority of draft tokens are accepted by the LLM. Otherwise, the draft cost may overcome the acceleration gain and lead to a negative speedup. To mitigate this, we propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work so that no effort is wasted on failed predictions. It has two key innovations. We configure the model layers as a pipeline in which early-exit (draft) computations and remaining-layer (verification) computations overlap. We interleave drafting and verification per token. While the LLM is verifying the current token in its final layers, the early-exit path simultaneously drafts the next token. Such a verify-while-draft scheme keeps all units busy and validates tokens on-the-fly analogous to pipelining the speculation and verification stages. Empirical results confirm that PPSD achieves state-of-the-art acceleration in self-speculative LLM inference. On diverse benchmarks, PPSD achieves speedup ratios in the range of 2.01x~3.81x, which gains almost the optimal acceleration at the fixed acceptance rate and exit position, showcasing its advancement in providing efficient self-speculation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 36 references that are in the map. First posted on arXiv 19 Sep 2025.
Builds on
31 of 36 references.
Closest ancestors · 18
Same direction, newest first
- 2025DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding8 Apr
- 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
- 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache5 Feb
- 2024QSpec: Speculative Decoding with Complementary Quantization Schemes15 Oct
- 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
- 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
- 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
- 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation16 Jul
- 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism6 Jun
- 2024Exploring and Improving Drafts in Blockwise Parallel Decoding14 Apr
- 2024Block Verification Accelerates Speculative Decoding15 Mar
- 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
6 more
- 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
- 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
- 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
- 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
- 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
- 2022Fast Inference from Transformers via Speculative Decoding30 Nov
Other references · 13
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2024Qwen2 Technical Reportcited by 2,120
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
- 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
- 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
- 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
- 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
- 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
- 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
- 2025PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
- 2025Self Speculative Decoding for Diffusion Large Language Models
- 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization
- 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
- 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
- 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference