# AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

arXiv: https://arxiv.org/abs/2501.02336 · HTML: https://mlsanity.com/paper/2501.02336/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 4 Jan 2025.

## Builds on (most cited first, 23 of 23)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [World Model on Million-Length Video And Language With Blockwise RingAttention](https://mlsanity.com/paper/2402.08268/) (2024)
- [InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory](https://mlsanity.com/paper/2402.04617/) (2024)
- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [Scaling Laws of RoPE-based Extrapolation](https://mlsanity.com/paper/2310.05209/) (2023)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference](https://mlsanity.com/paper/2307.02628/) (2023)
- [Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding](https://mlsanity.com/paper/2310.05424/) (2023)
- [Preble: Efficient Distributed Prompt Scheduling for LLM Serving](https://mlsanity.com/paper/2407.00023/) (2024)
- [EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism](https://mlsanity.com/paper/2312.04916/) (2023)
- [FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping](https://mlsanity.com/paper/2404.03865/) (2024)
- [Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy](https://mlsanity.com/paper/2404.06954/) (2024)
- [Hierarchical Skip Decoding for Efficient Autoregressive Text Generation](https://mlsanity.com/paper/2403.14919/) (2024)

## Built on by (most cited first, 12 of 12)

- [SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning](https://mlsanity.com/paper/2509.05614/) (2025)
- [From Prompts to Power: Measuring the Energy Footprint of LLM Inference](https://mlsanity.com/paper/2511.05597/) (2025)
- [DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policies](https://mlsanity.com/paper/2505.17420/) (2025)
- [DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding](https://mlsanity.com/paper/2504.19327/) (2025)
- [Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models](https://mlsanity.com/paper/2509.25584/) (2025)
- [LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference](https://mlsanity.com/paper/2601.02569/) (2026)
- [Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity](https://mlsanity.com/paper/2602.00397/) (2026)
- [River-LLM: Large Language Model Seamless Exit Based on KV Share](https://mlsanity.com/paper/2604.18396/) (2026)
- [QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs](https://mlsanity.com/paper/2602.10431/) (2026)
- [End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference](https://mlsanity.com/paper/2606.27743/) (2026)
- [CausalGate: Causal Importance Distillation for Transformer Module Pruning](https://mlsanity.com/paper/2607.22720/) (2026)
- [vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains](https://mlsanity.com/paper/2609.37062/) (2026)

## Abstract (arXiv)

Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping methods are promising optimizations but rarely explored in long-context inference. We observe that existing layer-wise skipping strategies have several limitations when applied in long-context inference, including the inability to adapt to model and context variability, disregard for sublayer significance, and inapplicability for the prefilling phase. This paper proposes sysname, an adaptive sublayer skipping method specifically designed for long-context inference. sysname adaptively identifies less important layers by leveraging on-the-fly similarity information, enables sublayer-wise skipping, and accelerates both the prefilling and decoding phases. The effectiveness of sysname is demonstrated through extensive experiments on various long-context benchmarks and models, showcasing its superior inference performance over existing baselines.