mlsanityData to 7 Oct 2026

Accelerating Large Language Model Inference with Self-Supervised Early Exits

arXiv 2407.21082 ↗First posted 30 Jul 2024cs.CL cs.LG stat.MLcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers. Each head is trained in a self-supervised manner to mimic the main model's predictions, allowing computation to stop early when a calibrated confidence threshold is reached. We evaluate several confidence metrics and show that entropy provides the most reliable separation between correct and incorrect predictions. Experiments on the Pythia model suite (70M to 2.8B parameters) demonstrate that our method significantly reduces inference cost while maintaining accuracy across multiple benchmarks. We further adapt this approach to speculative decoding, introducing Dynamic Self-Speculative Decoding (DSSD), which achieves 1.66x higher token acceptance than manually-tuned LayerSkip baselines with minimal hyperparameter tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 38 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

28 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding25 Apr

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
15 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  4. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  5. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  7. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  8. 2023A Survey on Model Compression for Large Language Modelscited by 263
  9. 2019Depth-Adaptive Transformercited by 227
  10. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  11. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  12. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  13. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  14. 2020Why should we add early exits to neural networks?cited by 70
  15. 2024EERO: Early Exit with Reject Option for Efficient Classification with limited budgetcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  2. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  3. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  4. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  5. 2026SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
  6. 2026SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
  7. 2026FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
  8. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization