mlsanityData to 7 Oct 2026

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

arXiv 2404.16710 ↗First posted 25 Apr 2024cs.CL cs.AI cs.LGcited by 302

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and checkpoints at https://github.com/facebookresearch/LayerSkip.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 62 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

47 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Confident Adaptive Language Modeling14 Jul

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
34 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2016Deep Networks with Stochastic Depthcited by 973
  10. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  13. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  14. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  15. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  16. 2023A Survey on Model Compression for Large Language Modelscited by 263
  17. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  18. 2019Depth-Adaptive Transformercited by 227
  19. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  20. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  21. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  22. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  23. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  24. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  25. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  26. 2022Formal Algorithms for Transformerscited by 70
  27. 2015Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognitioncited by 67
  28. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  29. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  30. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  31. 2019SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Modelscited by 22
  32. 2023Dropout Reduces Underfittingcited by 21
  33. 2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
  34. 2023A Performance Evaluation of a Quantized Large Language Model on Various Smartphonescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 116; 2026: 142.

302 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  6. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  7. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  8. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  9. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  10. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  11. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  12. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
38 more
  1. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  2. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  3. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  4. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  5. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  6. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  7. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  8. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  9. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  10. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  11. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  12. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  13. 2025AI Flow: Perspectives, Scenarios, and Approachescited by 44
  14. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  15. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  16. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 39
  17. 2024Faster Cascades via Speculative Decodingcited by 38
  18. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  19. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  20. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  21. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  22. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  23. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  24. 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
  25. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  26. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 23
  27. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  28. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  29. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  30. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  31. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  32. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  33. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  34. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  35. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  36. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  37. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 15
  38. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15

50 most cited of 302 All 302 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  2. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  3. 2024An Efficient Inference Framework for Early-exit Large Language Models
  4. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  5. 2023Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
  6. 2024FIRP: Faster LLM inference via future intermediate representation prediction
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  8. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism