# Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

arXiv: https://arxiv.org/abs/2402.02057 · HTML: https://mlsanity.com/paper/2402.02057/

## Key facts

- Cited by 298 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 54 of its 68 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)

## Built on by (most cited first, 30 of 298)

- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models](https://mlsanity.com/paper/2406.16838/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [The Mamba in the Llama: Distilling and Accelerating Hybrid Models](https://mlsanity.com/paper/2408.15237/) (2024)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [Accelerating Diffusion LLMs via Adaptive Parallel Decoding](https://mlsanity.com/paper/2506.00413/) (2025)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [Reward-Guided Speculative Decoding for Efficient LLM Reasoning](https://mlsanity.com/paper/2501.19324/) (2025)
- [PowerInfer-2: Fast Large Language Model Inference on a Smartphone](https://mlsanity.com/paper/2406.06282/) (2024)
- [TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding](https://mlsanity.com/paper/2404.11912/) (2024)
- [PEARL: Parallel Speculative Decoding with Adaptive Draft Length](https://mlsanity.com/paper/2408.11850/) (2024)
- [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024)
- [CLLMs: Consistency Large Language Models](https://mlsanity.com/paper/2403.00835/) (2024)
- [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024)
- [SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration](https://mlsanity.com/paper/2410.06916/) (2024)
- [OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure](https://mlsanity.com/paper/2406.17276/) (2024)
- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction](https://mlsanity.com/paper/2404.08509/) (2024)
- [Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding](https://mlsanity.com/paper/2410.01699/) (2024)
- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)
- [ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2502.00299/) (2025)
- [Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment](https://mlsanity.com/paper/2501.19309/) (2025)
- [SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning](https://mlsanity.com/paper/2504.07891/) (2025)
- [Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting](https://mlsanity.com/paper/2404.18911/) (2024)
- [Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion](https://mlsanity.com/paper/2408.05636/) (2024)
- [Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling](https://mlsanity.com/paper/2408.17355/) (2024)
- [OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts](https://mlsanity.com/paper/2503.22952/) (2025)

## Abstract (arXiv)

Autoregressive decoding of large language models (LLMs) is memory bandwidth bounded, resulting in high latency and significant wastes of the parallel processing power of modern accelerators. Existing methods for accelerating LLM decoding often require a draft model (e.g., speculative decoding), which is nontrivial to obtain and unable to generalize. In this paper, we introduce Lookahead decoding, an exact, parallel decoding algorithm that accelerates LLM decoding without needing auxiliary models or data stores. It allows trading per-step log(FLOPs) to reduce the number of total decoding steps, is more parallelizable on single or multiple modern accelerators, and is compatible with concurrent memory-efficient attention (e.g., FlashAttention). Our implementation of Lookahead decoding can speed up autoregressive decoding by up to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code completion tasks. Our code is avialable at https://github.com/hao-ai-lab/LookaheadDecoding