# On the Impact of Calibration Data in Post-training Quantization and Pruning

arXiv: https://arxiv.org/abs/2311.09755 · HTML: https://mlsanity.com/paper/2311.09755/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Intriguing Properties of Quantization at Scale](https://mlsanity.com/paper/2305.19268/) (2023)
- [Pruning vs Quantization: Which is Better?](https://mlsanity.com/paper/2307.02973/) (2023)

## Built on by (most cited first, 30 of 58)

- [Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models](https://mlsanity.com/paper/2504.04823/) (2025)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Reassessing Layer Pruning in LLMs: New Insights and Methods](https://mlsanity.com/paper/2411.15558/) (2024)
- [Beware of Calibration Data for Pruning Large Language Models](https://mlsanity.com/paper/2410.17711/) (2024)
- [ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks](https://mlsanity.com/paper/2312.08583/) (2023)
- [Self-calibration for Language Model Quantization and Pruning](https://mlsanity.com/paper/2410.17170/) (2024)
- [Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs](https://mlsanity.com/paper/2405.20835/) (2024)
- [Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox](https://mlsanity.com/paper/2406.12928/) (2024)
- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)
- [Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization](https://mlsanity.com/paper/2510.10618/) (2025)
- [Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression](https://mlsanity.com/paper/2504.07389/) (2025)
- [On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning](https://mlsanity.com/paper/2408.14398/) (2024)
- [NIRVANA: Structured Pruning Reimagined for Large Language Model Compression](https://mlsanity.com/paper/2509.14230/) (2025)
- [Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity](https://mlsanity.com/paper/2603.17354/) (2026)
- [Compressing Language Models for Specialized Domains](https://mlsanity.com/paper/2502.18424/) (2025)
- [Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models](https://mlsanity.com/paper/2508.18609/) (2025)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [Revisiting Large Language Model Pruning using Neuron Semantic Attribution](https://mlsanity.com/paper/2503.01542/) (2025)
- [μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts](https://mlsanity.com/paper/2505.18451/) (2025)
- [OneComp: One-Line Revolution for Generative AI Model Compression](https://mlsanity.com/paper/2603.28845/) (2026)
- [M-Wanda: Improving One-Shot Pruning for Multilingual LLMs](https://mlsanity.com/paper/2505.21171/) (2025)
- [Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions](https://mlsanity.com/paper/2508.17610/) (2025)
- [Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLM](https://mlsanity.com/paper/2601.18306/) (2026)
- [Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models](https://mlsanity.com/paper/2602.07899/) (2026)
- [Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs](https://mlsanity.com/paper/2608.08506/) (2026)
- [Certifying Compressed Language Models: An Audit and a Statistical Toolkit](https://mlsanity.com/paper/2608.15046/) (2026)
- [Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism](https://mlsanity.com/paper/2408.10473/) (2024)
- [Code-Driven Inductive Synthesis: Enhancing Reasoning Abilities of Large Language Models with Sequences](https://mlsanity.com/paper/2503.13109/) (2025)
- [CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding](https://mlsanity.com/paper/2511.19705/) (2025)
- [EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning](https://mlsanity.com/paper/2511.19935/) (2025)

## Abstract (arXiv)

Quantization and pruning form the foundation of compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated remarkable performance in a post-training setting. They rely upon calibration data, a small set of unlabeled examples that are used to generate layer activations. However, no prior work has systematically investigated how the calibration data impacts the effectiveness of model compression methods. In this paper, we present the first extensive empirical study on the effect of calibration data upon LLM performance. We trial a variety of quantization and pruning methods, datasets, tasks, and models. Surprisingly, we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness to the calibration data. Finally, we make a series of recommendations for the effective use of calibration data in LLM quantization and pruning.