mlsanityData to 14 Oct 2026

LaCo: Large Language Model Pruning via Layer Collapse

arXiv 2402.11187 ↗First posted 17 Feb 2024cs.CL cs.AIcited by 126

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) based on transformer are witnessing a notable trend of size expansion, which brings considerable costs to both model training and inference. However, existing methods such as model quantization, knowledge distillation, and model pruning are constrained by various issues, including hardware support limitations, the need for extensive training, and alterations to the model internal structure. In this paper, we propose a concise layer-wise structured pruner called textitLayer Collapse (LaCo), in which rear model layers collapse into a prior layer, enabling a rapid reduction in model size while preserving the model structure. Comprehensive experiments show that our method maintains an average task performance of over 80% at pruning ratios of 25-30%, significantly outperforming existing state-of-the-art structured pruning methods. We also conduct post-training experiments to confirm that the textitLaCo effectively inherits the parameters of the original model. Additionally, we perform ablation studies on various settings of textitLaCo. Finally, we discuss our motivation from the perspective of layer-wise similarity and evaluate the performance of the pruned LLMs across various pruning ratiosfootnoteurlhttps://github.com/yangyifei729/LaCo.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

31 of 41 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
19 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  6. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  9. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  12. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  13. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  14. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  15. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  16. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  17. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 81
  18. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  19. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 33; 2025: 50; 2026: 43.

126 papers.

Sort:

By area:

  1. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  2. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  3. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  4. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  5. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  6. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 45
  7. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 41
  8. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 36
  9. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 27
  10. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 27
  11. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  12. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 24
38 more
  1. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 19
  2. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 19
  3. 2025Towards AI Search Paradigmcited by 19
  4. 2025Instruction-Following Pruning for Large Language Modelscited by 17
  5. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 17
  6. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levelscited by 16
  7. 2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
  8. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 14
  9. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  10. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  11. 2024CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Informationcited by 11
  12. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  13. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  14. 2024FoldGPT: Simple and Effective Large Language Model Compression Schemecited by 9
  15. 2024Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancycited by 9
  16. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 9
  17. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  18. 2026Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruningcited by 8
  19. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 8
  20. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  21. 2026Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decodingcited by 7
  22. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  23. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  24. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  25. 2024GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inferencecited by 5
  26. 2024GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compressioncited by 5
  27. 2025Optimizing Small Language Models for In-Vehicle Function-Callingcited by 5
  28. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  29. 2025Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Modelscited by 5
  30. 2025FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compressioncited by 5
  31. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
  32. 2026Why Attention Patterns Exist: A Unifying Temporal Perspective Analysiscited by 5
  33. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  34. 2024Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?cited by 4
  35. 2024LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deploymentcited by 4
  36. 2025FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharingcited by 4
  37. 2025Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generationcited by 4
  38. 2025T'yr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimizationcited by 4

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
  2. 2024Streamlining Redundant Layers to Compress Large Language Models
  3. 2025Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
  4. 2024Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations
  5. 2025From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
  6. 2025MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
  7. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information
  8. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models