mlsanityData to 7 Oct 2026

Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods

arXiv 2402.02834 ↗First posted 5 Feb 2024cs.LG cs.CLcited by 80

Not on the map. Why?

Abstract (arXiv)

Structured pruning of modern large language models (LLMs) has emerged as a way of decreasing their high computational needs. Width pruning reduces the size of projection weight matrices (e.g., by removing attention heads) while maintaining the number of layers. Depth pruning, in contrast, removes entire layers or blocks, while keeping the size of the remaining weights unchanged. Most current research focuses on either width-only or a blend of width and depth pruning, with little comparative analysis between the two units (width vs. depth) concerning their impact on LLM inference efficiency. In this work, we show that simple depth pruning can effectively compress LLMs while achieving comparable or superior performance to recent width pruning studies. Our pruning method boosts inference speeds, especially under memory-constrained conditions that require limited batch sizes for running LLMs, where width pruning is ineffective. In retraining pruned models for quality recovery, continued pretraining on a large corpus markedly outperforms LoRA-based tuning, particularly at severe pruning ratios. We hope this work can help build compact yet capable LLMs. Code and models can be found at: https://github.com/Nota-NetsPresso/shortened-llm

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

49 of 69 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
37 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2019Are Sixteen Heads Really Better than One?cited by 803
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  13. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  14. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  15. 2019Importance Estimation for Neural Network Pruningcited by 531
  16. 2019The State of Sparsity in Deep Neural Networkscited by 471
  17. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  18. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  19. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  20. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  21. 2022Confident Adaptive Language Modelingcited by 297
  22. 2023A Survey on Model Compression for Large Language Modelscited by 263
  23. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  24. 2023DepGraph: Towards Any Structural Pruningcited by 202
  25. 2023Efficient Large Language Models: A Surveycited by 183
  26. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  27. 2021Block Pruning For Faster Transformerscited by 176
  28. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  29. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  30. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122
  31. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  32. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  33. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  34. 2022Unified Visual Transformer Compressioncited by 79
  35. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  36. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  37. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 36; 2026: 22.

80 papers.

Sort:

By area:

  1. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  2. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138
  3. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  5. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 44
  6. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  7. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  8. 2024A deeper look at depth pruning of LLMscited by 23
  9. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  10. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  11. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  12. 2024Superficial Safety Alignment Hypothesiscited by 14
38 more
  1. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  2. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  3. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  4. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  5. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  6. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  7. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  8. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 10
  9. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  10. 2024Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retrainingcited by 8
  11. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  12. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  13. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  14. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  15. 2024KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Modelscited by 6
  16. 2025AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Upcited by 6
  17. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  18. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
  19. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
  20. 2024Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Modelscited by 4
  21. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Modelscited by 4
  22. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  23. 2025Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Modelscited by 4
  24. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  25. 2025Choose Your Model Size: Any Compression of Large Language Models Without Re-Computationcited by 3
  26. 2025Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMscited by 3
  27. 2025GPTailor: Large Language Model Pruning Through Layer Cutting and Stitchingcited by 3
  28. 2026Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Modelscited by 3
  29. 2024Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Promptscited by 2
  30. 2024Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Trainingcited by 2
  31. 2024From Pruning to Grafting: Dynamic Knowledge Redistribution via Learnable Layer Fusioncited by 2
  32. 2025Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsitycited by 2
  33. 2025ZeroLM: Data-Free Transformer Architecture Search for Language Modelscited by 2
  34. 2025OverFill: Two-Stage Models for Efficient Language Model Decodingcited by 2
  35. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  36. 2025RCPU: Rotation-Constrained Error Compensation for Structured Pruning of Large Language Modelscited by 2
  37. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  38. 2026Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitionscited by 2

50 most cited of 80 All 80 (JSON)