mlsanityData to 7 Oct 2026

Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions

arXiv 2508.08192 ↗First posted 11 Aug 2025cs.CLcited by 7

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding is a standard method for accelerating the inference speed of large language models. However, scaling it for production environments poses several engineering challenges, including efficiently implementing different operations (e.g., tree attention and multi-round speculative decoding) on GPU. In this paper, we detail the training and inference optimization techniques that we have implemented to enable EAGLE-based speculative decoding at a production scale for Llama models. With these changes, we achieve a new state-of-the-art inference latency for Llama models. For example, Llama4 Maverick decodes at a speed of about 4 ms per token (with a batch size of one) on 8 NVIDIA H100 GPUs, which is 10% faster than the previously best known method. Furthermore, for EAGLE-based speculative decoding, our optimizations enable us to achieve a speed-up for large batch sizes between 1.4x and 2.0x at production scale.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 31 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

23 of 31 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  3. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
2 more
  1. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  2. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMscited by 5
  2. 2026Make Every Draft Count: Hidden State based Speculative Decodingcited by 4
  3. 2025HiSpec: Hierarchical Speculative Decoding for LLMscited by 3
  4. 2026Test-Time Speculationcited by 3
  5. 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluationcited by 1
  6. 2025SKYLENAGE Technical Report: Mathematical Reasoning and Contest-Innovation Benchmarks for Multi-Level Math Evaluationcited by 0
  7. 2025Direct Multi-Token Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  3. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  4. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding
  6. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
  7. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  8. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding