# PaSS: Parallel Speculative Sampling arXiv: https://arxiv.org/abs/2311.13581 ยท HTML: https://mlsanity.com/paper/2311.13581/ ## Key facts - Cited by 49 papers in the map (data through 14 Oct 2026). - Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. - Builds on 8 of its 11 references that are in the map. - First posted on arXiv 22 Nov 2023. ## Placed in - Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/) ## Builds on (most cited first, 8 of 8) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019) - [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017) - [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022) - [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018) ## Built on by (most cited first, 30 of 49) - [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024) - [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025) - [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024) - [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024) - [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024) - [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024) - [Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning](https://mlsanity.com/paper/2410.14157/) (2024) - [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024) - [Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential](https://mlsanity.com/paper/2507.11851/) (2025) - [Faster Cascades via Speculative Decoding](https://mlsanity.com/paper/2405.19261/) (2024) - [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025) - [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025) - [PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation](https://mlsanity.com/paper/2504.18583/) (2025) - [Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding](https://mlsanity.com/paper/2402.13720/) (2024) - [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025) - [ParallelSpec: Parallel Drafter for Efficient Speculative Decoding](https://mlsanity.com/paper/2410.05589/) (2024) - [Towards Optimal Multi-draft Speculative Decoding](https://mlsanity.com/paper/2502.18779/) (2025) - [Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference](https://mlsanity.com/paper/2405.18628/) (2024) - [Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation](https://mlsanity.com/paper/2411.18462/) (2024) - [Multi-Token Prediction Needs Registers](https://mlsanity.com/paper/2505.10518/) (2025) - [Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries](https://mlsanity.com/paper/2510.14751/) (2025) - [DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding](https://mlsanity.com/paper/2505.19201/) (2025) - [Continuous Latent Diffusion Language Model](https://mlsanity.com/paper/2605.06548/) (2026) - [BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models](https://mlsanity.com/paper/2401.12522/) (2024) - [Fast Large Language Model Collaborative Decoding via Speculation](https://mlsanity.com/paper/2502.01662/) (2025) - [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025) - [Semformer: Transformer Language Models with Semantic Planning](https://mlsanity.com/paper/2409.11143/) (2024) - [Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding](https://mlsanity.com/paper/2509.11961/) (2025) - [The Mystery of the Pathological Path-star Task for Language Models](https://mlsanity.com/paper/2410.13779/) (2024) - [Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models](https://mlsanity.com/paper/2410.20418/) (2024) ## Abstract (arXiv) Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressively, requiring a forward pass for each generated token, and thus reading the full set of parameters from memory. This memory access forms the primary bottleneck for generation and it worsens as the model size increases. Moreover, executing a forward pass for multiple tokens in parallel often takes nearly the same time as it does for just one token. These two observations lead to the development of speculative sampling, where a second smaller model is used to draft a few tokens, that are then validated or rejected using a single forward pass of the large model. Unfortunately, this method requires two models that share the same tokenizer and thus limits its adoption. As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model. Our approach only requires an additional input token that marks the words that will be generated simultaneously. We show promising performance (up to 30% speed-up) while requiring only as few as O(demb) additional parameters.