# Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study

arXiv: https://arxiv.org/abs/2307.08072 · HTML: https://mlsanity.com/paper/2307.08072/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)
- [ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation](https://mlsanity.com/paper/2303.08302/) (2023)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [Compressing deep neural networks by matrix product operators](https://mlsanity.com/paper/1904.06194/) (2019)
- [A Model Compression Method with Matrix Product Operators for Speech Enhancement](https://mlsanity.com/paper/2010.04950/) (2020)

## Built on by (most cited first, 30 of 43)

- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [Evaluating Quantized Large Language Models](https://mlsanity.com/paper/2402.18158/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [A Comprehensive Evaluation of Quantization Strategies for Large Language Models](https://mlsanity.com/paper/2402.16775/) (2024)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [How Does Quantization Affect Multilingual LLMs?](https://mlsanity.com/paper/2407.03211/) (2024)
- [Accurate LoRA-Finetuning Quantization of LLMs via Information Retention](https://mlsanity.com/paper/2402.05445/) (2024)
- [Mixture Compressor for Mixture-of-Experts LLMs Gains More](https://mlsanity.com/paper/2410.06270/) (2024)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization](https://mlsanity.com/paper/2509.23202/) (2025)
- [LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL](https://mlsanity.com/paper/2503.18596/) (2025)
- [DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs](https://mlsanity.com/paper/2312.05215/) (2023)
- [Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance](https://mlsanity.com/paper/2402.12819/) (2024)
- [Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations](https://mlsanity.com/paper/2504.06792/) (2025)
- [PrivateLoRA For Efficient Privacy Preserving LLM](https://mlsanity.com/paper/2311.14030/) (2023)
- [Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant](https://mlsanity.com/paper/2409.11055/) (2024)
- [Exploring the Mystery of Influential Data for Mathematical Reasoning](https://mlsanity.com/paper/2404.01067/) (2024)
- [Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox](https://mlsanity.com/paper/2406.12928/) (2024)
- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)
- [Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability](https://mlsanity.com/paper/2506.15629/) (2025)
- [Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs](https://mlsanity.com/paper/2508.19432/) (2025)
- [Good Questions Help Zero-Shot Image Reasoning](https://mlsanity.com/paper/2312.01598/) (2023)
- [PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models](https://mlsanity.com/paper/2502.13179/) (2025)
- [Chatmap : Large Language Model Interaction with Cartographic Data](https://mlsanity.com/paper/2310.01429/) (2023)
- [Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression](https://mlsanity.com/paper/2405.12591/) (2024)
- [Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates](https://mlsanity.com/paper/2408.12263/) (2024)
- [Importance-Aware Data Selection for Efficient LLM Instruction Tuning](https://mlsanity.com/paper/2511.07074/) (2025)
- [When Quantization Affects Confidence of Large Language Models?](https://mlsanity.com/paper/2405.00632/) (2024)
- [Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluation](https://mlsanity.com/paper/2407.13069/) (2024)

## Abstract (arXiv)

Despite the superior performance, Large Language Models~(LLMs) require significant computational resources for deployment and use. To overcome this issue, quantization methods have been widely applied to reduce the memory footprint of LLMs as well as increasing the inference rate. However, a major challenge is that low-bit quantization methods often lead to performance degradation. It is important to understand how quantization impacts the capacity of LLMs. Different from previous studies focused on overall performance, this work aims to investigate the impact of quantization on emergent abilities, which are important characteristics that distinguish LLMs from small language models. Specially, we examine the abilities of in-context learning, chain-of-thought reasoning, and instruction-following in quantized LLMs. Our empirical experiments show that these emergent abilities still exist in 4-bit quantization models, while 2-bit models encounter severe performance degradation on the test of these abilities. To improve the performance of low-bit models, we conduct two special experiments: (1) fine-gained impact analysis that studies which components (or substructures) are more sensitive to quantization, and (2) performance compensation through model fine-tuning. Our work derives a series of important findings to understand the impact of quantization on emergent abilities, and sheds lights on the possibilities of extremely low-bit quantization for LLMs.