# Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy

arXiv: https://arxiv.org/abs/2404.06954 · HTML: https://mlsanity.com/paper/2404.06954/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference](https://mlsanity.com/paper/2307.02628/) (2023)
- [DiffuSum: Generation Enhanced Extractive Summarization with Diffusion](https://mlsanity.com/paper/2305.01735/) (2023)
- [Anytime Dense Prediction with Confidence Adaptivity](https://mlsanity.com/paper/2104.00749/) (2021)
- [Synchronous Bidirectional Inference for Neural Sequence Generation](https://mlsanity.com/paper/1902.08955/) (2019)
- [Probabilistic Adaptive Computation Time](https://mlsanity.com/paper/1712.00386/) (2017)

## Built on by (most cited first, 17 of 17)

- [SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration](https://mlsanity.com/paper/2410.06916/) (2024)
- [Understanding Layer Significance in LLM Alignment](https://mlsanity.com/paper/2410.17875/) (2024)
- [AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference](https://mlsanity.com/paper/2501.02336/) (2025)
- [Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules](https://mlsanity.com/paper/2407.06677/) (2024)
- [KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization](https://mlsanity.com/paper/2505.16162/) (2025)
- [Dynamic layer selection in decoder-only transformers](https://mlsanity.com/paper/2410.20022/) (2024)
- [DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policies](https://mlsanity.com/paper/2505.17420/) (2025)
- [ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models](https://mlsanity.com/paper/2508.17892/) (2025)
- [IG-Pruning: Input-Guided Block Pruning for Large Language Models](https://mlsanity.com/paper/2511.02213/) (2025)
- [Distributed Collaborative Inference System in Next-Generation Networks and Communication](https://mlsanity.com/paper/2412.12102/) (2024)
- [FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction](https://mlsanity.com/paper/2410.12513/) (2024)
- [LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning](https://mlsanity.com/paper/2510.14211/) (2025)
- [TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference](https://mlsanity.com/paper/2603.21365/) (2026)
- [Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning](https://mlsanity.com/paper/2606.03113/) (2026)
- [CausalGate: Causal Importance Distillation for Transformer Module Pruning](https://mlsanity.com/paper/2607.22720/) (2026)
- [Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference](https://mlsanity.com/paper/2609.08189/) (2026)
- [vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains](https://mlsanity.com/paper/2609.37062/) (2026)

## Abstract (arXiv)

Recently, dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations through elaborate heuristics or additional predictors. However, in the decoding process of existing approaches, different samples are assigned different computational budgets, which cannot guarantee a stable and precise acceleration effect. Furthermore, existing approaches generally skip multiple contiguous layers at the bottom or top of the layers, leading to a drastic change in the model's layer-wise representations, and thus a consequent performance degeneration. Therefore, we propose a Unified Layer Skipping strategy, which selects the number of layers to skip computation based solely on the target speedup ratio, and then skips the corresponding number of intermediate layer computations in a balanced manner. Since the Unified Layer Skipping strategy is independent of input samples, it naturally supports popular acceleration techniques such as batch decoding and KV caching, thus demonstrating more practicality for real-world applications. Experimental results on two common tasks, i.e., machine translation and text summarization, indicate that given a target speedup ratio, the Unified Layer Skipping strategy significantly enhances both the inference performance and the actual model throughput over existing dynamic approaches.