FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
On the map: Efficient LLM inference and serving → Early exit and edge inference
Abstract (arXiv)
Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference. FlexEE makes early exiting practical for LLM decoding through layer-wise exit supervision for reliable intermediate-layer prediction, self-speculative decoding over a Top-K local vocabulary for low-cost exit decisions, and dynamic hidden state management for KV-cache-correct and memory-aware execution. Across generative and downstream tasks, FlexEE enables efficient early exit with minimal accuracy degradation, delivering up to 1.27×/3.16× and 1.25×/2.83× end-to-end speedups on Llama2-7B and Llama3-8B under 0%/50% weight offloading, respectively.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 60 references that are in the map. First posted on arXiv 15 Sep 2026.
Builds on
43 of 60 references.
Closest ancestors · 2
Same direction, newest first
- 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding25 Apr
- 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism8 Dec
Other references · 41
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Pointer Sentinel Mixture Modelscited by 3,096
29 more
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
- 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
- 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
- 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
- 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
- 2019Depth-Adaptive Transformercited by 236
- 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
- 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
- 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
- 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 82
- 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
- 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025HAMburger: Accelerating LLM Inference via Token Smashing
- 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
- 2024An Efficient Inference Framework for Early-exit Large Language Models
- 2026River-LLM: Large Language Model Seamless Exit Based on KV Share
- 2023Accelerating LLM Inference with Staged Speculative Decoding
- 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
- 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
- 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference