mlsanityData to 14 Oct 2026

EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

arXiv 2406.16858 ↗First posted 24 Jun 2024cs.CL cs.LGcited by 370

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Inference with modern Large Language Models (LLMs) is expensive and time-consuming, and speculative sampling has proven to be an effective solution. Most speculative sampling methods such as EAGLE use a static draft tree, implicitly assuming that the acceptance rate of draft tokens depends only on their position. Interestingly, we found that the acceptance rate of draft tokens is also context-dependent. In this paper, building upon EAGLE, we propose EAGLE-2, which introduces a new technique of context-aware dynamic draft tree into drafting modeling. This improvement leverages the fact that the draft model of EAGLE is well-calibrated: the confidence scores from the draft model approximate acceptance rates with small errors. We conducted extensive evaluations on three series of LLMs and six tasks, with EAGLE-2 achieving speedup ratios 3.05x-4.26x, which is 20%-40% faster than EAGLE-1. EAGLE-2 also ensures that the distribution of the generated text remains unchanged, making it a lossless acceleration algorithm.

Key facts. Cited by 370 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 57 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

44 of 57 references.

Closest ancestors · 25

Same direction, newest first

  1. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  2. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  3. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  4. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  5. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Models14 Mar
  6. 2024CLLMs: Consistency Large Language Models28 Feb
  7. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  8. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding19 Feb
  9. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  10. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  11. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
13 more
  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  3. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  4. 2023PaSS: Parallel Speculative Sampling22 Nov
  5. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  7. 2023Online Speculative Decoding11 Oct
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  9. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  10. 2023Speculative Decoding with Big Little Decoder15 Feb
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  13. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 19

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2019The State of Sparsity in Deep Neural Networkscited by 476
  8. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  9. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
7 more
  1. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  2. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  3. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  4. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 65
  5. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  6. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  7. 2023Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inferencecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 27; 2025: 142; 2026: 201.

370 papers.

Sort:

By area:

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  2. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 135
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  4. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  5. 2025Efficient Reasoning with Hidden Thinkingcited by 75
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 72
  7. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  8. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  9. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  10. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  11. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 47
  12. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 47
38 more
  1. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 47
  2. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  3. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 39
  4. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  5. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  6. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 37
  7. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  8. 2026Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decodingcited by 34
  9. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 32
  10. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  11. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  12. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 29
  13. 2026Accelerating Speculative Decoding with Block Diffusion Draft Treescited by 29
  14. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  15. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  16. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  17. 2026DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inferencecited by 24
  18. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  19. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  20. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 22
  21. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  22. 2024Dynamic Depth Decoding: Faster Speculative Decoding for LLMscited by 20
  23. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  24. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 20
  25. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 20
  26. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 19
  27. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  28. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  29. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structurescited by 18
  30. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  31. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  32. 2025StereoVLA: Enhancing Vision-Language-Action Models with Stereo Visioncited by 18
  33. 2024Mixture of Attentions For Speculative Decodingcited by 17
  34. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 17
  35. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 16
  36. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 16
  37. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 16
  38. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 15

50 most cited of 370 All 370 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models
  2. 2026Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
  3. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
  4. 2025Towards Optimal Multi-draft Speculative Decoding
  5. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  6. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability
  7. 2024FastDraft: How to Train Your Draft
  8. 2025Hierarchical Verification of Speculative Beams for Accelerating LLM Inference