mlsanityData to 7 Oct 2026

Dynamic layer selection in decoder-only transformers

arXiv 2410.20022 ↗First posted 26 Oct 2024cs.CL cs.LGcited by 3

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

The vast size of Large Language Models (LLMs) has prompted a search to optimize inference. One effective approach is dynamic inference, which adapts the architecture to the sample-at-hand to reduce the overall computational cost. We empirically examine two common dynamic inference methods for natural language generation (NLG): layer skipping and early exiting. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping, as opposed to early exit. We demonstrate the difficulty of using hidden state information to adapt computation on a per-token basis for layer skipping. Finally, we show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains by constructing an oracle controller. Remarkably, we find that there exists an allocation which achieves equal performance to the full model using only 23.3% of its layers on average.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 28 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

24 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference19 Dec
  2. 2022Confident Adaptive Language Modeling14 Jul

Other references · 22

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2024Mixtral of Expertscited by 1,785
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  9. 2021Dynamic Neural Networks: A Surveycited by 405
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  11. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  12. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
10 more
  1. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  2. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  3. 2019Depth-Adaptive Transformercited by 227
  4. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  5. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  6. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  7. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  8. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  9. 2023Learning to Skip for Language Modelingcited by 19
  10. 2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 9
  2. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  3. 2026The Diminishing Returns of Early-Exit Decoding in Modern LLMscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Large Language Model Inference via Early-Exiting Algorithms
  2. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  3. 2025DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policies
  4. 2026Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding
  5. 2026Thinking Before Constraining: A Unified Decoding Framework for Large Language Models
  6. 2026ADEPT: Adaptive Dynamic Early-Exit Process for Transformers
  7. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
  8. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMs