mlsanityData to 14 Oct 2026

Beware of Calibration Data for Pruning Large Language Models

arXiv 2410.17711 ↗First posted 23 Oct 2024cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. Post-training pruning is a promising method that does not require resource-intensive iterative training and only needs a small amount of calibration data to assess the importance of parameters. Recent research has enhanced post-training pruning from different aspects but few of them systematically explore the effects of calibration data, and it is unclear if there exist better calibration data construction strategies. We fill this blank and surprisingly observe that calibration data is also crucial to post-training pruning, especially for high sparsity. Through controlled experiments on important influence factors of calibration data, including the pruning settings, the amount of data, and its similarity with pre-training data, we observe that a small size of data is adequate, and more similar data to its pre-training stage can yield better performance. As pre-training data is usually inaccessible for advanced LLMs, we further provide a self-generating calibration data synthesis strategy to construct feasible calibration data. Experimental results on recent strong open-source LLMs (e.g., DCLM, and LLaMA-3) show that the proposed strategy can enhance the performance of strong pruning methods (e.g., Wanda, DSnoT, OWL) by a large margin (up to 2.68%). Code is available at https://github.com/Dereck0602/calibrationdata.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 43 of its 51 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

43 of 51 references.

References · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
31 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  2. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  5. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  8. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  9. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  11. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  12. 2021Block Pruning For Faster Transformerscited by 177
  13. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  14. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  15. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  16. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
  17. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
  18. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  19. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  20. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 58
  21. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  22. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  23. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  24. 2023Dynamic Sparse Training with Structured Sparsitycited by 38
  25. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  26. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  27. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
  28. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 10
  29. 2024Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimizationcited by 8
  30. 2022Pruning Pre-trained Language Models Without Fine-Tuningcited by 7
  31. 2024Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  2. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  3. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  4. 2025RSQ: Learning from Important Tokens Leads to Better Quantized LLMscited by 3
  5. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Modelscited by 3
  6. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2
  7. 2025A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?cited by 1
  8. 2025EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruningcited by 1
  9. 2026GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Modelscited by 1
  10. 2026Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantizationcited by 1
  11. 2026OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruningcited by 1
  12. 2026Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Modelscited by 0
3 more
  1. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0
  2. 2026No Free Efficiency: Revisiting the Trade-off Between Training Efficiency and Model Vulnerabilitycited by 0
  3. 2026QK-Wanda: Coupling Queries and Keys for Unstructured Pruningcited by 0