mlsanityData to 7 Oct 2026

PaSS: Parallel Speculative Sampling

arXiv 2311.13581 ↗First posted 22 Nov 2023cs.CLcited by 49

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressively, requiring a forward pass for each generated token, and thus reading the full set of parameters from memory. This memory access forms the primary bottleneck for generation and it worsens as the model size increases. Moreover, executing a forward pass for multiple tokens in parallel often takes nearly the same time as it does for just one token. These two observations lead to the development of speculative sampling, where a second smaller model is used to draft a few tokens, that are then validated or rejected using a single forward pass of the large model. Unfortunately, this method requires two models that share the same tokenizer and thus limits its adoption. As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model. Our approach only requires an additional input token that marks the words that will be generated simultaneously. We show promising performance (up to 30% speed-up) while requiring only as few as O(demb) additional parameters.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 8 of its 11 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

8 of 11 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 6

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2017Non-Autoregressive Neural Machine Translationcited by 457
  5. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 20; 2026: 6.

49 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2024The pitfalls of next-token predictioncited by 156
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  8. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  9. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  10. 2024Faster Cascades via Speculative Decodingcited by 38
  11. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  12. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
37 more
  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  2. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  4. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  5. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  6. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  7. 2025Multi-Token Prediction Needs Registerscited by 16
  8. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 15
  9. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 14
  10. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  11. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  12. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12
  13. 2026Continuous Latent Diffusion Language Modelcited by 12
  14. 2025A Survey on Parallel Reasoningcited by 11
  15. 2024Semformer: Transformer Language Models with Semantic Planningcited by 9
  16. 2024The Mystery of the Pathological Path-star Task for Language Modelscited by 8
  17. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8
  18. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Modelscited by 7
  19. 2024Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 7
  20. 2024Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokenscited by 6
  21. 2025Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoEcited by 6
  22. 2024Exploring and Improving Drafts in Blockwise Parallel Decodingcited by 5
  23. 2024BlockLLM: Multi-tenant Finer-grained Serving for Large Language Modelscited by 5
  24. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  25. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decodingcited by 4
  26. 2024BLens: Contrastive Captioning of Binary Functions using Ensemble Embeddingcited by 3
  27. 2024COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinementcited by 3
  28. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  29. 2025Reinforcement Speculative Decoding for Fast Rankingcited by 3
  30. 2026The Coupling Within: Flow Matching via Distilled Normalizing Flowscited by 2
  31. 2026The State-Prediction Separation Hypothesiscited by 2
  32. 2024CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decodingcited by 1
  33. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 1
  34. 2025Cross-Attention Speculative Decodingcited by 0
  35. 2025S4C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Modelscited by 0
  36. 2026Efficient Training-Free Multi-Token Prediction via Embedding-Space Probingcited by 0
  37. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion
  2. 2024Multi-Candidate Speculative Decoding
  3. 2026Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
  4. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
  5. 2025Speculative Decoding Speed-of-Light: Optimal Lower Bounds via Branching Random Walks
  6. 2024Decoding Speculative Decoding
  7. 2026Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
  8. 2025Parallel Token Prediction for Language Models