mlsanityData to 7 Oct 2026

EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

arXiv 2503.01840 ↗First posted 3 Mar 2025cs.CLcited by 318

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The sequential nature of modern LLMs makes them expensive and slow, and speculative sampling has proven to be an effective solution to this problem. Methods like EAGLE perform autoregression at the feature level, reusing top-layer features from the target model to achieve better results than vanilla speculative sampling. A growing trend in the LLM community is scaling up training data to improve model intelligence without increasing inference costs. However, we observe that scaling up data provides limited improvements for EAGLE. We identify that this limitation arises from EAGLE's feature prediction constraints. In this paper, we introduce EAGLE-3, which abandons feature prediction in favor of direct token prediction and replaces reliance on top-layer features with multi-layer feature fusion via a technique named training-time test. These improvements significantly enhance performance and enable the draft model to fully benefit from scaling up training data. Our experiments include both chat models and reasoning models, evaluated on five tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with about 1.4x improvement over EAGLE-2. In the SGLang framework, EAGLE-3 achieves a 1.38x throughput improvement at a batch size of 64. The code is available at https://github.com/SafeAILab/EAGLE.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 47 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

39 of 47 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree17 Dec
  2. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  4. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  5. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  6. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  7. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  8. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding19 Feb
  9. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  10. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  11. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
7 more
  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  3. 2023PaSS: Parallel Speculative Sampling22 Nov
  4. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  5. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  6. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  7. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
8 more
  1. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  2. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  3. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  4. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  5. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  6. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  7. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  8. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 87; 2026: 231.

318 papers.

Sort:

By area:

  1. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  2. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  3. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  4. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  5. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  6. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  7. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 31
  8. 2026Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decodingcited by 31
  9. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  10. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  11. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  12. 2026Accelerating Speculative Decoding with Block Diffusion Draft Treescited by 27
38 more
  1. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  2. 2026DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inferencecited by 23
  3. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  4. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  5. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  6. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  7. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  8. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 17
  9. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  10. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  11. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 14
  12. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  13. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  14. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  15. 2025Scaling Speculative Decoding with Lookahead Reasoningcited by 13
  16. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
  17. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  18. 2025Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generationcited by 13
  19. 2025Speculative Decoding: Performance or Illusion?cited by 13
  20. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 13
  21. 2026Speculative Speculative Decodingcited by 13
  22. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 12
  23. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 12
  24. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 12
  25. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 12
  26. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMscited by 12
  27. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Draftingcited by 12
  28. 2025A Survey on Parallel Reasoningcited by 11
  29. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 11
  30. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  31. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11
  32. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 10
  33. 2026Draft-OPD: On-Policy Distillation for Speculative Draft Modelscited by 10
  34. 2025Faster and Better LLMs via Latency-Aware Test-Time Scalingcited by 9
  35. 2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMscited by 9
  36. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  37. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 9
  38. 2025SpecExit: Accelerating Large Reasoning Model via Speculative Exitcited by 9

50 most cited of 318 All 318 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  2. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
  3. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  4. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  5. 2024Accelerating Production LLMs with Combined Token/Embedding Speculators
  6. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
  7. 2026Flatter Tokens are More Valuable for Speculative Draft Model Training
  8. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs