mlsanityData to 7 Oct 2026

Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding

arXiv 2307.05908 ↗First posted 12 Jul 2023cs.CL cs.LGcited by 40

Not on the map. Why?

Cited by 32 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents "Predictive Pipelined Decoding (PPD)," an approach that speeds up greedy decoding in Large Language Models (LLMs) while maintaining the exact same output as the original decoding. Unlike conventional strategies, PPD employs additional compute resources to parallelize the initiation of subsequent token decoding during the current token decoding. This method reduces decoding latency and reshapes the understanding of trade-offs in LLM decoding strategies. We have developed a theoretical framework that allows us to analyze the trade-off between computation and latency. Using this framework, we can analytically estimate the potential reduction in latency associated with our proposed method, achieved through the assessment of the match rate, represented as p_correct. The results demonstrate that the use of extra computational resources has the potential to accelerate LLM decoding. Additionally, we implement PPD and conduct preliminary experiments to empirically validate its efficacy, addressing potential practical overheads not covered by theoretical analysis.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

41 of 52 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
29 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2018Hierarchical Neural Story Generationcited by 1,182
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2015Teaching Machines to Read and Comprehendcited by 1,039
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  10. 2019The State of Sparsity in Deep Neural Networkscited by 471
  11. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  12. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  13. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  14. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  15. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  16. 2022Confident Adaptive Language Modelingcited by 297
  17. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  18. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  19. 2023Speculative Decoding with Big Little Decodercited by 163
  20. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  21. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  22. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  23. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  24. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  25. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  26. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  27. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  28. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  29. 2023To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiencycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 22; 2025: 14; 2026: 3.

40 papers.

Sort:

By area:

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  6. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  8. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  9. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  10. 2024Fast On-device LLM Inference with NPUscited by 58
  11. 2024Decoding Speculative Decodingcited by 33
  12. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
28 more
  1. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  2. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  3. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
  4. 2024PLD+: Accelerating LLM inference by leveraging Language Model Artifactscited by 16
  5. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculationcited by 14
  6. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Accelerationcited by 14
  7. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  8. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  9. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  10. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  11. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  12. 2024Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokenscited by 6
  13. 2024KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learningcited by 6
  14. 2025CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Qualitycited by 6
  15. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
  16. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
  17. 2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
  18. 2024Efficient Sparse Attention needs Adaptive Token Releasecited by 4
  19. 2024Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Modelscited by 4
  20. 2025Reinforcement Speculative Decoding for Fast Rankingcited by 3
  21. 2025Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problemcited by 3
  22. 2026Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal Statescited by 3
  23. 2025Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performancecited by 2
  24. 2025Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulationscited by 2
  25. 2026PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Lengthcited by 2
  26. 2024CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decodingcited by 1
  27. 2025S4C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Modelscited by 0
  28. 2026VIA-SD: Verification via Intra-Model Routing for Speculative Decodingcited by 0