mlsanityData to 7 Oct 2026

vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

arXiv 2609.37062 ↗First posted 29 Sep 2026cs.DC cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely on specialized generation loops and do not integrate with modern serving engines. As a result, fewer executed layers do not necessarily translate into lower serving latency: FlexiDepth skips 8 of Llama-3-8B's 32 layers on average, yet its standard generation loop decodes 14.6--21.0% more slowly than the base model. We present vSkipper, a virtualization layer that makes dynamic layer skippers pluggable in serving engines while preserving continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs. At each routed layer, vSkipper groups tokens by the skipper's decision and uses routed execution only when predicted to be profitable. We implement vSkipper in SGLang and evaluate the released FlexiDepth checkpoint against upstream SGLang under identical prompts, arrivals, output lengths, and launch settings. At the knee of upstream's load curve, vSkipper reduces mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH. Under saturation, it increases request throughput by 11.3% and 7.4%. Serving adds no statistically resolved quality loss beyond the checkpoint's own. Across synthetic skip policies, two Qwen3 skippers, and three GPUs, we demonstrate reuse without workload-specific tuning. To our knowledge, vSkipper is the first system to realize serving-efficiency gains from per-token interior layer skipping within a modern LLM serving engine. The code is open-sourced as an SGLang fork at https://github.com/AKafakA/sglang-vskipper/tree/vskipper-ref

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 30 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

22 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2018CoQA: A Conversational Question Answering Challengecited by 674
  7. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  9. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  10. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  11. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  12. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
7 more
  1. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  2. 2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17
  3. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  4. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  5. 2025DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policiescited by 3
  6. 2025Dynamic Rebatching for Efficient Early-Exit Inference with DREXcited by 2
  7. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  2. 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
  3. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
  4. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
  5. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters
  6. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  7. 2026LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
  8. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving