mlsanityData to 7 Oct 2026

The Diminishing Returns of Early-Exit Decoding in Modern LLMs

arXiv 2603.23701 ↗First posted 24 Mar 2026cs.CL cs.AIcited by 1

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

In Large Language Model (LLM) inference, early-exit refers to stopping computation at an intermediate layer once the prediction is sufficiently confident, thereby reducing latency and cost. However, recent LLMs adopt improved pretraining recipes and architectures that reduce layer redundancy, potentially limiting early-exit opportunities. We re-evaluate layer-wise early-exit in modern LLMs and analyze how intermediate representations evolve during training. We introduce a metric to quantify a model's intrinsic suitability for early-exit and propose a benchmark for researchers to explore the potential early-exit benefits on different models and workloads. Our results show a diminishing trend in early-exit effectiveness across newer model generations. We further find that dense transformers generally offer greater early-exit potential than Mixture-of-Experts and State Space Models. In addition, larger models, particularly those with more than 20 billion parameters, and base pretrained models without specialized tuning tend to exhibit higher early-exit potential.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 44 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

29 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Dynamic layer selection in decoder-only transformers26 Oct
  2. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding25 Apr
  3. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism8 Dec

Other references · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  10. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
14 more
  1. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  2. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  3. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  4. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  5. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  6. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  7. 2025Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoningcited by 36
  8. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 31
  9. 2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
  10. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  11. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  12. 2025Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspectivecited by 13
  13. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 9
  14. 2025Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decodingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
  2. 2024An Efficient Inference Framework for Early-exit Large Language Models
  3. 2026River-LLM: Large Language Model Seamless Exit Based on KV Share
  4. 2024Dynamic Vocabulary Pruning in Early-Exit LLMs
  5. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  6. 2025SimLens for Early Exit in Large Language Models: Eliciting Accurate Latent Predictions with One More Token
  7. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
  8. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models