mlsanityData to 7 Oct 2026

Streamlining Redundant Layers to Compress Large Language Models

arXiv 2403.19135 ↗First posted 28 Mar 2024cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned.LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency.Our code is available at https://github.com/RUCKBReasoning/LLM-Streamline

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 17.

32 papers.

Sort:

By area:

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  2. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  3. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  4. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  5. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  6. 2025Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Trainingcited by 9
  7. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  8. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  9. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  10. 2026On the Limits of Layer Pruning for Generative Reasoning in Large Language Modelscited by 3
  11. 2025Boomerang Distillation Enables Zero-Shot Model Size Interpolationcited by 2
  12. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiencycited by 2
20 more
  1. 2026Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initializationcited by 2
  2. 2026High-Fidelity Pruning for Large Language Modelscited by 1
  3. 2026Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMscited by 1
  4. 2026Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Thinkcited by 1
  5. 2026TACTICL: Task-Aware Compression of Tabular ICL Modelscited by 1
  6. 2025Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsitycited by 0
  7. 2025Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Onescited by 0
  8. 2025LOP: Learning Optimal Pruning for Efficient On-Demand MLLMs Scalingcited by 0
  9. 2025COMPACT: Common-token Optimized Model Pruning Across Channels and Tokenscited by 0
  10. 2025INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Modelscited by 0
  11. 2026Compressing LLMs with MoP: Mixture of Prunerscited by 0
  12. 2026FlattenGPT: Depth Compression for Transformer with Layer Flatteningcited by 0
  13. 2026GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensationcited by 0
  14. 2026A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Modelscited by 0
  15. 2026Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcyclingcited by 0
  16. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0
  17. 2026Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMscited by 0
  18. 2026XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compressioncited by 0
  19. 2026Forward-Free LLM Depth Pruning via Weight Redundancycited by 0
  20. 2026Does the VGGT Family Need All Its Layers?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pruning Laws for Large Language Models
  2. 2025MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
  3. 2023Just CHOP: Embarrassingly Simple LLM Compression
  4. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
  5. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information
  6. 2024Pruning Foundation Models for High Accuracy without Retraining
  7. 2026From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
  8. 2024A deeper look at depth pruning of LLMs