mlsanityData to 14 Oct 2026

ParallelSpec: Parallel Drafter for Efficient Speculative Decoding

arXiv 2410.05589 ↗First posted 8 Oct 2024cs.CL cs.LGcited by 22

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding has proven to be an efficient solution to large language model (LLM) inference, where the small drafter predicts future tokens at a low cost, and the target model is leveraged to verify them in parallel. However, most existing works still draft tokens auto-regressively to maintain sequential dependency in language modeling, which we consider a huge computational burden in speculative decoding. We present ParallelSpec, an alternative to auto-regressive drafting strategies in state-of-the-art speculative decoding approaches. In contrast to auto-regressive drafting in the speculative stage, we train a parallel drafter to serve as an efficient speculative model. ParallelSpec learns to efficiently predict multiple future tokens in parallel using a single model, and it can be integrated into any speculative decoding framework that requires aligning the output distributions of the drafter and the target model with minimal training cost. Experimental results show that ParallelSpec accelerates baseline methods in latency up to 62% on text generation benchmarks from different domains, and it achieves 2.84X overall speedup on the Llama-2-13B model using third-party evaluation criteria.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 60 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

52 of 60 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  2. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  3. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  6. 2024Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration18 Apr
  7. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  8. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding21 Feb
  9. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  10. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding19 Feb
  11. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  12. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
12 more
  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  5. 2023PaSS: Parallel Speculative Sampling22 Nov
  6. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  7. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  9. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
16 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  5. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  6. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  7. 2022Confident Adaptive Language Modelingcited by 310
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  9. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  10. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  11. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 77
  12. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  13. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 65
  14. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  15. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
  16. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 13.

22 papers.

Sort:

By area:

  1. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 47
  2. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 29
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  4. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 10
  5. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 10
  6. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  7. 2026PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decodingcited by 8
  8. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  9. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 6
  10. 2026SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Draftingcited by 3
  11. 2026PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Lengthcited by 2
  12. 2025Polybasic Speculative Decoding Through a Theoretical Perspectivecited by 1
10 more
  1. 2026MineDraft: A Framework for Batch Parallel Speculative Decodingcited by 1
  2. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  3. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decodingcited by 1
  4. 2025Cross-Attention Speculative Decodingcited by 0
  5. 2025Fast Inference via Hierarchical Speculative Decodingcited by 0
  6. 2026Diffusion Language Model Parallel Decoding via Product-of-Experts Bridgecited by 0
  7. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel draftercited by 0
  8. 2026Recovering Off-Policy Supervision for Speculative Decodingcited by 0
  9. 2026SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Modelscited by 0
  10. 2026JetSpec++: Foresight Parallel Drafting with Self Refinementscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
  2. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  3. 2026Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
  4. 2026Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
  5. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  6. 2024Decoding Speculative Decoding
  7. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  8. 2026Speculative Decoding with a Speculative Vocabulary