# Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models

arXiv: https://arxiv.org/abs/2405.04304 · HTML: https://mlsanity.com/paper/2405.04304/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 7 May 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Speculative Streaming: Fast LLM Inference without Auxiliary Models](https://mlsanity.com/paper/2402.11131/) (2024)
- [Tandem Transformers for Inference Efficient LLMs](https://mlsanity.com/paper/2402.08644/) (2024)

## Built on by (most cited first, 30 of 31)

- [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024)
- [AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability](https://mlsanity.com/paper/2410.18351/) (2024)
- [Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies](https://mlsanity.com/paper/2502.05202/) (2025)
- [AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures](https://mlsanity.com/paper/2412.18910/) (2024)
- [LLM-42: Enabling Determinism in LLM Inference with Verified Speculation](https://mlsanity.com/paper/2601.17768/) (2026)
- [Speeding up Speculative Decoding via Sequential Approximate Verification](https://mlsanity.com/paper/2502.04557/) (2025)
- [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](https://mlsanity.com/paper/2511.14102/) (2025)
- [Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference](https://mlsanity.com/paper/2405.14105/) (2024)
- [AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up](https://mlsanity.com/paper/2505.24584/) (2025)
- [Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding](https://mlsanity.com/paper/2605.20104/) (2026)
- [D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding](https://mlsanity.com/paper/2607.14647/) (2026)
- [Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs](https://mlsanity.com/paper/2506.05387/) (2025)
- [OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding](https://mlsanity.com/paper/2507.02659/) (2025)
- [DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification](https://mlsanity.com/paper/2604.07622/) (2026)
- [AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding](https://mlsanity.com/paper/2607.25852/) (2026)
- [DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving](https://mlsanity.com/paper/2509.01083/) (2025)
- [PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length](https://mlsanity.com/paper/2602.01274/) (2026)
- [TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts](https://mlsanity.com/paper/2608.22788/) (2026)
- [SMART: When is it Actually Worth Expanding a Speculative Tree?](https://mlsanity.com/paper/2604.09731/) (2026)
- [Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding](https://mlsanity.com/paper/2609.02897/) (2026)
- [Automatic Task Detection and Heterogeneous LLM Speculative Decoding](https://mlsanity.com/paper/2505.08600/) (2025)
- [Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing](https://mlsanity.com/paper/2605.14978/) (2026)
- [An Interpretable Latency Model for Speculative Decoding in LLM Serving](https://mlsanity.com/paper/2605.15051/) (2026)
- [Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection](https://mlsanity.com/paper/2607.05148/) (2026)
- [LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization](https://mlsanity.com/paper/2608.08721/) (2026)
- [AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines](https://mlsanity.com/paper/2610.01108/) (2026)
- [Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation](https://mlsanity.com/paper/2610.05978/) (2026)
- [Trajectory-Retrieval Speculative Decoding: When Does a Model's Own History Help?](https://mlsanity.com/paper/2610.07350/) (2026)
- [DLoop: Looped Speculative Decoding](https://mlsanity.com/paper/2610.07659/) (2026)
- [APEX: Speculate smarter, not deeper](https://mlsanity.com/paper/2610.07780/) (2026)

## Abstract (arXiv)

Speculative decoding is commonly used for reducing the inference latency of large language models. Its effectiveness depends highly on the speculation lookahead (SL)-the number of tokens generated by the draft model at each iteration. In this work we show that the common practice of using the same SL for all iterations (static SL) is suboptimal. We introduce DISCO (DynamIc SpeCulation lookahead Optimization), a novel method for dynamically selecting the SL. Our experiments with four datasets show that DISCO reaches an average speedup of 10% compared to the best static SL baseline, while generating the exact same text.