mlsanityData to 7 Oct 2026

EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

arXiv 2401.15077 ↗First posted 26 Jan 2024cs.LG cs.CLcited by 564

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Autoregressive decoding makes the inference of Large Language Models (LLMs) time-consuming. In this paper, we reconsider speculative sampling and derive two key observations. Firstly, autoregression at the feature (second-to-top-layer) level is more straightforward than at the token level. Secondly, the inherent uncertainty in feature (second-to-top-layer) level autoregression constrains its performance. Based on these insights, we introduce EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency), a simple yet highly efficient speculative sampling framework. By incorporating a token sequence advanced by one time step, EAGLE effectively resolves the uncertainty, enabling precise second-to-top-layer feature prediction with minimal overhead. We conducted comprehensive evaluations of EAGLE, including all models from the Vicuna and LLaMA2-Chat series, the MoE model Mixtral 8x7B Instruct, and tasks in dialogue, code generation, mathematical reasoning, and instruction following. For LLaMA2-Chat 70B, EAGLE achieved a latency speedup ratio of 2.7x-3.5x, doubled throughput, while maintaining the distribution of the generated text.

Key facts. Cited by 564 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 36 of its 42 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

36 of 42 references.

References · 36

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
24 more
  1. 2019The State of Sparsity in Deep Neural Networkscited by 471
  2. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  3. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  4. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  5. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  6. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  7. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  8. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  9. 2023Speculative Decoding with Big Little Decodercited by 163
  10. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  11. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  12. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  13. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  14. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  15. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  16. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  17. 2023Online Speculative Decodingcited by 109
  18. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  19. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  20. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  21. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  22. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  23. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  24. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 80; 2025: 200; 2026: 284.

564 papers.

Sort:

By area:

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2026Gemma 4 Technical Reportcited by 187
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  7. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  8. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  9. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  10. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  11. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  12. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 99
38 more
  1. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  2. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 93
  3. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  4. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  5. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  6. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  7. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  8. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  9. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  10. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  11. 2024Parallelized Autoregressive Visual Generationcited by 58
  12. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  13. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  14. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  15. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  16. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  17. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  18. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  19. 2024Faster Cascades via Speculative Decodingcited by 38
  20. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  21. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  22. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 36
  23. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  24. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  25. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  26. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  27. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  28. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  29. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  30. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 31
  31. 2026Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decodingcited by 31
  32. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  33. 2024Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Modelscited by 29
  34. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  35. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  36. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  37. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  38. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27

50 most cited of 564 All 564 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  2. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2024When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  5. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  6. 2024Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation
  7. 2024Decoding Speculative Decoding
  8. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models