# Self-calibration for Language Model Quantization and Pruning

arXiv: https://arxiv.org/abs/2410.17170 · HTML: https://mlsanity.com/paper/2410.17170/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 81 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 30 of 51)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)

## Built on by (most cited first, 13 of 13)

- [Compressing Language Models for Specialized Domains](https://mlsanity.com/paper/2502.18424/) (2025)
- [Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models](https://mlsanity.com/paper/2511.18864/) (2025)
- [Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLM](https://mlsanity.com/paper/2601.18306/) (2026)
- [Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization](https://mlsanity.com/paper/2607.00908/) (2026)
- [On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain](https://mlsanity.com/paper/2607.01444/) (2026)
- [Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models](https://mlsanity.com/paper/2608.21019/) (2026)
- [Fair-GPTQ: Bias-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2509.15206/) (2025)
- [Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels](https://mlsanity.com/paper/2604.24008/) (2026)
- [Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage](https://mlsanity.com/paper/2606.03328/) (2026)
- [Damage Predicts Recovery: When Calibration Data Matters in Compressing Financial LLMs](https://mlsanity.com/paper/2609.26241/) (2026)
- [FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization](https://mlsanity.com/paper/2610.14934/) (2026)
- [RECAP: Calibrating Low-Bit Reasoning Models on the History They Read](https://mlsanity.com/paper/2610.15259/) (2026)
- [Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization](https://mlsanity.com/paper/2610.20353/) (2026)

## Abstract (arXiv)

Quantization and pruning are fundamental approaches for model compression, enabling efficient inference for language models. In a post-training setting, state-of-the-art quantization and pruning methods require calibration data, a small set of unlabeled examples. Conventionally, this is randomly sampled web text, aiming to reflect the model training data. However, this poses two key problems: (1) unrepresentative calibration examples can harm model performance, and (2) organizations increasingly avoid releasing model training data. In this paper, we propose self-calibration as a solution. Our approach requires no external data, instead leveraging the model itself to generate synthetic calibration data, with a view to better approximating the pre-training data distribution. We extensively compare the performance of self-calibration with several baselines, across a variety of models, compression methods, and tasks. Our approach proves consistently competitive in maximizing downstream task performance, frequently outperforming even using real data.