mlsanityData to 7 Oct 2026

LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

arXiv 2609.13682 ↗First posted 12 Sep 2026cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with a lightweight per-layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters), training both jointly under a gate-regularized language-modeling objective. Across 10 independently-seeded training runs, LayerRoute converges to an identical skip-pattern structure in every run - a consistent set of 9 middle layers (8-16) becomes skip-eligible in all 10 seeds - and delivers genuine, verified wallclock speedup in every run (1.02x-1.06x, mean 1.04x). Quality is preserved or improved in every configuration tested: joint LoRA adaptation yields a perplexity improvement over the unmodified backbone in all 10 seeds (mean delta = -1.16 and -1.11 across the two evaluation splits used). We further verify the router performs genuine, non-trivial per-input computation: gate decisions in skip-eligible layers change the actual skip/run outcome for 87-100% of held-out samples, confirming real input-dependent routing rather than a fixed pruning pattern. LayerRoute trains in under 7 minutes on a single A100 and adds negligible overhead beyond the routing decision itself. We report our full reproducibility methodology, including a systematic diagnostic investigation into what determines the router's per-input decisions, as part of this work.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 38 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

32 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Confident Adaptive Language Modeling14 Jul

Other references · 31

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
19 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  9. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  10. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  11. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  13. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  14. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  15. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  16. 2019Depth-Adaptive Transformercited by 227
  17. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  18. 2021PonderNet: Learning to Pondercited by 156
  19. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives
  2. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
  3. 2024FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction
  4. 2026Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
  5. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  6. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
  7. 2025What Layers When: Learning to Skip Compute in LLMs with Residual Gates
  8. 2026LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models