mlsanityData to 7 Oct 2026

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

arXiv 2403.03853 ↗First posted 6 Mar 2024cs.CLcited by 369

Not on the map. Why?

Cited by 35 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. However, in this study, we discovered that many layers of LLMs exhibit high similarity, and some layers play a negligible role in network functionality. Based on this observation, we define a metric called Block Influence (BI) to gauge the significance of each layer in LLMs. We then propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers in LLMs based on their BI scores. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT is orthogonal to quantization-like methods, enabling further reduction in parameters and computation. The ability to achieve better results through simple layer removal, as opposed to more complex pruning techniques, suggests a high degree of redundancy in the model architecture.

Key facts. Cited by 369 papers in the map (data through 7 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

46 of 58 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
34 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2019Root Mean Square Layer Normalizationcited by 1,425
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  13. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  15. 2018CoQA: A Conversational Question Answering Challengecited by 674
  16. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  17. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  18. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  19. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  20. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  21. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  22. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  23. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  24. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  25. 2021Post-Training Quantization for Vision Transformercited by 274
  26. 2023A Survey on Model Compression for Large Language Modelscited by 263
  27. 2020Understanding the Difficulty of Training Transformerscited by 210
  28. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  29. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  30. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  31. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  32. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  33. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  34. 2021A New Measure of Model Redundancy for Compressed Convolutional Neural Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 62; 2025: 146; 2026: 161.

369 papers.

Sort:

By area:

  1. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  2. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  3. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  4. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  5. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  6. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  7. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  8. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  9. 2025The Curse of Depth in Large Language Modelscited by 58
  10. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  11. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  12. 2024Transformer Layers as Painterscited by 50
38 more
  1. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  4. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  5. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  6. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 44
  7. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  8. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
  9. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  10. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  11. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  12. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 35
  13. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 35
  14. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  15. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  16. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  17. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 26
  18. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 26
  19. 2024SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Modelscited by 24
  20. 2024Llama Guard 3-1B-INT4: Compact and Efficient Safeguard for Human-AI Conversationscited by 24
  21. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  22. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  23. 2024A deeper look at depth pruning of LLMscited by 23
  24. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  25. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  26. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  27. 2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
  28. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  29. 2025RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformerscited by 18
  30. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  31. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  32. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  33. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levelscited by 16
  34. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
  35. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  36. 2025Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakerscited by 16
  37. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  38. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15

50 most cited of 369 All 369 (JSON)