mlsanityData to 7 Oct 2026

Adaptive Layer-skipping in Pre-trained LLMs

arXiv 2503.23798 ↗First posted 31 Mar 2025cs.CL cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Various layer-skipping methods have been proposed to accelerate token generation in large language models (LLMs). However, limited attention has been paid to a fundamental question: How do computational demands vary across the generation of different tokens? In this work, we introduce FlexiDepth, a method that dynamically adjusts the number of Transformer layers used in text generation. By incorporating a plug-in router and adapter, FlexiDepth enables adaptive computation in LLMs without modifying their original parameters. Applied to Llama-3-8B, it skips 8 out of 32 layers while maintaining full benchmark performance. Our experiments reveal that computational demands in LLMs significantly vary based on token type. Specifically, generating repetitive tokens or fixed phrases requires fewer layers, whereas producing tokens involving computation or high uncertainty requires more layers. Despite the computational savings, FlexiDepth does not yet achieve wall-clock speedup due to varied skipping patterns and I/O overhead. To inspire future work and advance research on practical speedup, we open-sourced FlexiDepth and a dataset documenting its layer allocation patterns.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

37 of 46 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
25 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  3. 2018CoQA: A Conversational Question Answering Challengecited by 674
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  7. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  8. 2022Confident Adaptive Language Modelingcited by 297
  9. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  12. 2021Learned Token Pruning for Transformerscited by 138
  13. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  14. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  15. 2024What Matters in Transformers? Not All Attention is Neededcited by 83
  16. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  17. 2024Transformer Layers as Painterscited by 50
  18. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
  19. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  20. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 26
  21. 2023Learning to Skip for Language Modelingcited by 19
  22. 2023Sparse Backpropagation for MoE Trainingcited by 16
  23. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  24. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8
  25. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 14.

24 papers.

Sort:

By area:

  1. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
  2. 2025Dr.LLM: Dynamic Layer Routing in LLMscited by 10
  3. 2026DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulationcited by 8
  4. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  5. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 6
  6. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 6
  7. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 5
  8. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  9. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Modelscited by 3
  10. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understandingcited by 2
  11. 2025Dynamic Rebatching for Efficient Early-Exit Inference with DREXcited by 2
  12. 2026Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decodingcited by 2
12 more
  1. 2025Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Modelscited by 1
  2. 2026MACRO: Markov Chain Routing of Transformer Layerscited by 1
  3. 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
  4. 2025Direct Multi-Token Decodingcited by 0
  5. 2025LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoningcited by 0
  6. 2026BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inferencecited by 0
  7. 2026End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inferencecited by 0
  8. 2026CausalGate: Causal Importance Distillation for Transformer Module Pruningcited by 0
  9. 2026Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inferencecited by 0
  10. 2026Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exitcited by 0
  11. 2026When Less Compute Is More: Adaptive Early Exit Improves Pretrained Outlier Detectioncited by 0
  12. 2026vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gainscited by 0