mlsanityData to 13 Oct 2026

SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks

arXiv 2402.09025 ↗First posted 14 Feb 2024cs.CL cs.LGcited by 96

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have proven to be highly effective across various natural language processing tasks. However, their large number of parameters poses significant challenges for practical deployment. Pruning, a technique aimed at reducing the size and complexity of LLMs, offers a potential solution by removing redundant components from the network. Despite the promise of pruning, existing methods often struggle to achieve substantial end-to-end LLM inference speedup. In this paper, we introduce SLEB, a novel approach designed to streamline LLMs by eliminating redundant transformer blocks. We choose the transformer block as the fundamental unit for pruning, because LLMs exhibit block-level redundancy with high similarity between the outputs of neighboring blocks. This choice allows us to effectively enhance the processing speed of LLMs. Our experimental results demonstrate that SLEB outperforms previous LLM pruning methods in accelerating LLM inference while also maintaining superior perplexity and accuracy, making SLEB as a promising technique for enhancing the efficiency of LLMs. The code is available at: https://github.com/jiwonsong-dev/SLEB.

Key facts. Cited by 96 papers in the map (data through 13 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

24 of 36 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  7. 2016Pointer Sentinel Mixture Modelscited by 3,047
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,017
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
12 more
  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 787
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 646
  3. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 381
  4. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 318
  5. 2022Confident Adaptive Language Modelingcited by 302
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 250
  7. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  8. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  9. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  10. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  11. 2020SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inferencecited by 30
  12. 2020Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Formatcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 13; 2025: 47; 2026: 36.

96 papers.

Sort:

By area:

  1. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  2. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  3. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  4. 2024EvoPress: Accurate Dynamic Model Compression via Evolutionary Searchcited by 20
  5. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  6. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  7. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  8. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  9. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  10. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  11. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  12. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
38 more
  1. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  2. 2025Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMscited by 8
  3. 2025DipSVD: Dual-importance Protected SVD for Efficient LLM Compressioncited by 8
  4. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  5. 2024LPZero: Language Model Zero-cost Proxy Search from Zerocited by 7
  6. 2024Skipping Computations in Multimodal LLMscited by 7
  7. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  8. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layerscited by 7
  9. 2026Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruningcited by 7
  10. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  11. 2025SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Modelscited by 6
  12. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  13. 2025Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensationcited by 5
  14. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
  15. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  16. 2025Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Modelscited by 4
  17. 2025Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generationcited by 4
  18. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  19. 2025Pruning Large Language Models by Identifying and Preserving Functional Networkscited by 4
  20. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  21. 2024GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compressioncited by 3
  22. 2025DReSS: Data-driven Regularized Structured Streamlining for Large Language Modelscited by 3
  23. 2025Prompt-based Depth Pruning of Large Language Modelscited by 3
  24. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  25. 2025Two-Stage Regularization-Based Structured Pruning for LLMscited by 3
  26. 2025PruneCD: Contrasting Pruned Self Model to Improve Decoding Factualitycited by 3
  27. 2026On the Limits of Layer Pruning for Generative Reasoning in Large Language Modelscited by 3
  28. 2026Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Modelscited by 3
  29. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 3
  30. 2026Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?cited by 3
  31. 2024P2 Law: Scaling Law for Post-Training After Model Pruningcited by 2
  32. 2024From Pruning to Grafting: Dynamic Knowledge Redistribution via Learnable Layer Fusioncited by 2
  33. 2025EvoP: Robust LLM Inference via Evolutionary Pruningcited by 2
  34. 2025Position-Aware Depth Decay Decoding (D3): Boosting Large Language Model Inference Efficiencycited by 2
  35. 2025ZeroLM: Data-Free Transformer Architecture Search for Language Modelscited by 2
  36. 2025ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformationscited by 2
  37. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Informationcited by 2
  38. 2025Less Redundancy: Boosting Practicality of Vision Language Model in Walking Assistantscited by 2

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging the true depth of LLMs
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  3. 2026Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy
  4. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
  5. 2024Streamlining Redundant Layers to Compress Large Language Models
  6. 2025SPAP: Structured Pruning via Alternating Optimization and Penalty Methods
  7. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
  8. 2024A Survey on Efficient Inference for Large Language Models