mlsanityData to 7 Oct 2026

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

arXiv 2402.02057 ↗First posted 3 Feb 2024cs.LG cs.CLcited by 295

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive decoding of large language models (LLMs) is memory bandwidth bounded, resulting in high latency and significant wastes of the parallel processing power of modern accelerators. Existing methods for accelerating LLM decoding often require a draft model (e.g., speculative decoding), which is nontrivial to obtain and unable to generalize. In this paper, we introduce Lookahead decoding, an exact, parallel decoding algorithm that accelerates LLM decoding without needing auxiliary models or data stores. It allows trading per-step log(FLOPs) to reduce the number of total decoding steps, is more parallelizable on single or multiple modern accelerators, and is compatible with concurrent memory-efficient attention (e.g., FlashAttention). Our implementation of Lookahead decoding can speed up autoregressive decoding by up to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code completion tasks. Our code is avialable at https://github.com/hao-ai-lab/LookaheadDecoding

Key facts. Cited by 295 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 54 of its 68 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

54 of 68 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  5. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  6. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  7. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  8. 2023Online Speculative Decoding11 Oct
  9. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  10. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  12. 2023Speculative Decoding with Big Little Decoder15 Feb
3 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2016Layer Normalizationcited by 5,579
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2023Mistral 7Bcited by 3,065
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
27 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  12. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  13. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  14. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  15. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  16. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  17. 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 51
  18. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  19. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  20. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  21. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  22. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  23. 2023Generating Summaries with Controllable Readability Levelscited by 24
  24. 2023Adaptive and Personalized Exercise Generation for Online Language Learningcited by 18
  25. 2022Universal Conditional Masked Language Pre-training for Neural Machine Translationcited by 12
  26. 2020ProphetNet-Ads: A Looking Ahead Strategy for Generative Retrieval Models in Sponsored Search Enginecited by 11
  27. 2013Lookahead Strategies for Sequential Monte Carlocited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 78; 2025: 115; 2026: 102.

295 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  5. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  6. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  7. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  8. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  9. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  10. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  11. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  12. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 99
38 more
  1. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  2. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 83
  3. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  4. 2024CLLMs: Consistency Large Language Modelscited by 71
  5. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  6. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  7. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  8. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  9. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  10. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  11. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  12. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  13. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  14. 2024Fast On-device LLM Inference with NPUscited by 58
  15. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  16. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  17. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  18. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  19. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  20. 2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 46
  21. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  22. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  23. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  24. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  25. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  26. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  27. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 37
  28. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 36
  29. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  30. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  31. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  32. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  33. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  34. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
  35. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 26
  36. 2024Non-myopic Generation of Language Models for Reasoning and Planningcited by 25
  37. 2024On Speculative Decoding for Multimodal Large Language Modelscited by 24
  38. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23

50 most cited of 295 All 295 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  2. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  3. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  4. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models
  5. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding
  6. 2023Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster
  7. 2023Efficient LLM inference solution on Intel GPU
  8. 2026Faster LLM Inference via Sequential Monte Carlo