# Streamlining Redundant Layers to Compress Large Language Models

arXiv: https://arxiv.org/abs/2403.19135 · HTML: https://mlsanity.com/paper/2403.19135/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2024.

## Built on by (most cited first, 30 of 35)

- [HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://mlsanity.com/paper/2411.01433/) (2024)
- [SAM Decoding: Speculative Decoding via Suffix Automaton](https://mlsanity.com/paper/2411.10666/) (2024)
- [What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective](https://mlsanity.com/paper/2410.23743/) (2024)
- [ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers](https://mlsanity.com/paper/2504.00502/) (2025)
- [PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference](https://mlsanity.com/paper/2509.04467/) (2025)
- [Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training](https://mlsanity.com/paper/2506.22638/) (2025)
- [Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization](https://mlsanity.com/paper/2510.10618/) (2025)
- [RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding](https://mlsanity.com/paper/2505.22135/) (2025)
- [Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation](https://mlsanity.com/paper/2510.15304/) (2025)
- [TACTICL: Task-Aware Compression of Tabular ICL Models](https://mlsanity.com/paper/2608.10837/) (2026)
- [On the Limits of Layer Pruning for Generative Reasoning in Large Language Models](https://mlsanity.com/paper/2602.01997/) (2026)
- [Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency](https://mlsanity.com/paper/2604.24380/) (2026)
- [Boomerang Distillation Enables Zero-Shot Model Size Interpolation](https://mlsanity.com/paper/2510.05064/) (2025)
- [FlattenGPT: Depth Compression for Transformer with Layer Flattening](https://mlsanity.com/paper/2602.08858/) (2026)
- [Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs](https://mlsanity.com/paper/2605.15491/) (2026)
- [Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization](https://mlsanity.com/paper/2605.29327/) (2026)
- [GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation](https://mlsanity.com/paper/2602.14649/) (2026)
- [High-Fidelity Pruning for Large Language Models](https://mlsanity.com/paper/2603.08083/) (2026)
- [Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Think](https://mlsanity.com/paper/2606.20246/) (2026)
- [Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity](https://mlsanity.com/paper/2506.00932/) (2025)
- [Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones](https://mlsanity.com/paper/2506.05641/) (2025)
- [LOP: Learning Optimal Pruning for Efficient On-Demand MLLMs Scaling](https://mlsanity.com/paper/2506.12826/) (2025)
- [COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens](https://mlsanity.com/paper/2509.06836/) (2025)
- [INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models](https://mlsanity.com/paper/2511.19676/) (2025)
- [Compressing LLMs with MoP: Mixture of Pruners](https://mlsanity.com/paper/2602.06127/) (2026)
- [A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models](https://mlsanity.com/paper/2605.25344/) (2026)
- [Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling](https://mlsanity.com/paper/2605.26496/) (2026)
- [Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization](https://mlsanity.com/paper/2608.12953/) (2026)
- [Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs](https://mlsanity.com/paper/2608.22854/) (2026)
- [XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression](https://mlsanity.com/paper/2609.02083/) (2026)

## Abstract (arXiv)

This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned.LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency.Our code is available at https://github.com/RUCKBReasoning/LLM-Streamline