# SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting

arXiv: https://arxiv.org/abs/2504.08850 · HTML: https://mlsanity.com/paper/2504.08850/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 25 of 25)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models](https://mlsanity.com/paper/2408.11393/) (2024)
- [DRS: Dynamic Resource Scheduling for Real-Time Analytics over Fast Streams](https://mlsanity.com/paper/1501.03610/) (2015)

## Built on by (most cited first, 23 of 23)

- [HoliTom: Holistic Token Merging for Fast Video Large Language Models](https://mlsanity.com/paper/2505.21334/) (2025)
- [SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning](https://mlsanity.com/paper/2509.05614/) (2025)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulation](https://mlsanity.com/paper/2602.22896/) (2026)
- [V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval](https://mlsanity.com/paper/2512.12284/) (2025)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editing](https://mlsanity.com/paper/2509.22244/) (2025)
- [Make Every Draft Count: Hidden State based Speculative Decoding](https://mlsanity.com/paper/2602.21224/) (2026)
- [GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question Answering](https://mlsanity.com/paper/2604.06794/) (2026)
- [Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2509.10570/) (2025)
- [SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation](https://mlsanity.com/paper/2509.13848/) (2025)
- [Dynamic Rebatching for Efficient Early-Exit Inference with DREX](https://mlsanity.com/paper/2512.15705/) (2025)
- [SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning](https://mlsanity.com/paper/2603.23483/) (2026)
- [The Diminishing Returns of Early-Exit Decoding in Modern LLMs](https://mlsanity.com/paper/2603.23701/) (2026)
- [River-LLM: Large Language Model Seamless Exit Based on KV Share](https://mlsanity.com/paper/2604.18396/) (2026)
- [FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving](https://mlsanity.com/paper/2604.20503/) (2026)
- [NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies](https://mlsanity.com/paper/2605.26444/) (2026)
- [Synera: Synergistic LLM Serving across Device and Cloud at Scale](https://mlsanity.com/paper/2511.07423/) (2025)
- [TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference](https://mlsanity.com/paper/2603.21365/) (2026)
- [MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models](https://mlsanity.com/paper/2607.22586/) (2026)
- [FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference](https://mlsanity.com/paper/2609.17008/) (2026)
- [DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory](https://mlsanity.com/paper/2609.34380/) (2026)
- [Speculative Safety Honeypot: Toward Proactive Defense Against Multi-turn Agent Attacks](https://mlsanity.com/paper/2609.39549/) (2026)

## Abstract (arXiv)

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access. In this paper, we present SpecEE, a fast LLM inference engine with speculative early exiting. (1) At the algorithm level, we propose the speculation-based lightweight predictor design by exploiting the probabilistic correlation between the speculative tokens and the correct results and high parallelism of GPUs. (2) At the system level, we point out that not all layers need a predictor and design the two-level heuristic predictor scheduling engine based on skewed distribution and contextual similarity. (3) At the mapping level, we point out that different decoding methods share the same essential characteristics, and propose the context-aware merged mapping for predictor with efficient GPU implementations to support speculative decoding, and form a framework for various existing orthogonal acceleration techniques (e.g., quantization and sparse activation) on cloud and personal computer (PC) scenarios, successfully pushing the Pareto frontier of accuracy and speedup. It is worth noting that SpecEE can be applied to any LLM by negligible training overhead in advance without affecting the model original parameters. Extensive experiments show that SpecEE achieves 2.25x and 2.43x speedup with Llama2-7B on cloud and PC scenarios respectively.