mlsanityData to 14 Oct 2026

SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting

arXiv 2504.08850 ↗First posted 11 Apr 2025cs.DC cs.AIcited by 23

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access. In this paper, we present SpecEE, a fast LLM inference engine with speculative early exiting. (1) At the algorithm level, we propose the speculation-based lightweight predictor design by exploiting the probabilistic correlation between the speculative tokens and the correct results and high parallelism of GPUs. (2) At the system level, we point out that not all layers need a predictor and design the two-level heuristic predictor scheduling engine based on skewed distribution and contextual similarity. (3) At the mapping level, we point out that different decoding methods share the same essential characteristics, and propose the context-aware merged mapping for predictor with efficient GPU implementations to support speculative decoding, and form a framework for various existing orthogonal acceleration techniques (e.g., quantization and sparse activation) on cloud and personal computer (PC) scenarios, successfully pushing the Pareto frontier of accuracy and speedup. It is worth noting that SpecEE can be applied to any LLM by negligible training overhead in advance without affecting the model original parameters. Extensive experiments show that SpecEE achieves 2.25x and 2.43x speedup with Llama2-7B on cloud and PC scenarios respectively.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 46 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

25 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  4. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb

Other references · 21

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2024A Survey on Large Language Models for Code Generationcited by 658
  12. 2021Dynamic Neural Networks: A Surveycited by 408
9 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  2. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  6. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  7. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 75
  8. 2024First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Modelscited by 6
  9. 2015DRS: Dynamic Resource Scheduling for Real-Time Analytics over Fast Streamscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  2. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 43
  3. 2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
  4. 2026DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulationcited by 8
  5. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 5
  6. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  7. 2025FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editingcited by 4
  8. 2026Make Every Draft Count: Hidden State based Speculative Decodingcited by 4
  9. 2026GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answeringcited by 4
  10. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  11. 2025SpecDiff: Accelerating Diffusion Model Inference with Self-Speculationcited by 2
  12. 2025Dynamic Rebatching for Efficient Early-Exit Inference with DREXcited by 2
11 more
  1. 2026SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planningcited by 1
  2. 2026The Diminishing Returns of Early-Exit Decoding in Modern LLMscited by 1
  3. 2026River-LLM: Large Language Model Seamless Exit Based on KV Sharecited by 1
  4. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  5. 2026NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabulariescited by 1
  6. 2025Synera: Synergistic LLM Serving across Device and Cloud at Scalecited by 0
  7. 2026TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inferencecited by 0
  8. 2026MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Modelscited by 0
  9. 2026FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inferencecited by 0
  10. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memorycited by 0
  11. 2026Speculative Safety Honeypot: Toward Proactive Defense Against Multi-turn Agent Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models
  2. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  3. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  4. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  5. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models
  6. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts
  7. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding