# What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation

arXiv: https://arxiv.org/abs/2403.06408 · HTML: https://mlsanity.com/paper/2403.06408/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation](https://mlsanity.com/paper/2303.08302/) (2023)
- [ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats](https://mlsanity.com/paper/2307.09782/) (2023)
- [Extreme Compression for Pre-trained Transformers Made Simple and Efficient](https://mlsanity.com/paper/2206.01859/) (2022)

## Built on by (most cited first, 23 of 23)

- [How Does Quantization Affect Multilingual LLMs?](https://mlsanity.com/paper/2407.03211/) (2024)
- [Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization](https://mlsanity.com/paper/2509.23202/) (2025)
- [Perturbation-Restrained Sequential Model Editing](https://mlsanity.com/paper/2405.16821/) (2024)
- [Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging](https://mlsanity.com/paper/2406.16330/) (2024)
- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)
- [Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models](https://mlsanity.com/paper/2409.00084/) (2024)
- [Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluation](https://mlsanity.com/paper/2407.13069/) (2024)
- [Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining](https://mlsanity.com/paper/2504.13932/) (2025)
- [Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation](https://mlsanity.com/paper/2506.22776/) (2025)
- [Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization](https://mlsanity.com/paper/2602.24059/) (2026)
- [The Uneven Impact of Post-Training Quantization in Machine Translation](https://mlsanity.com/paper/2508.20893/) (2025)
- [SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV](https://mlsanity.com/paper/2509.25853/) (2025)
- [NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2602.06694/) (2026)
- [RUQuant: Towards Refining Uniform Quantization for Large Language Models](https://mlsanity.com/paper/2604.04013/) (2026)
- [Towards Joint Quantization and Token Pruning of Vision-Language Models](https://mlsanity.com/paper/2604.17320/) (2026)
- [UltraSketchLLM: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators](https://mlsanity.com/paper/2506.17255/) (2025)
- [Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature](https://mlsanity.com/paper/2511.03261/) (2025)
- [Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2601.07878/) (2026)
- [K-Quantization and its Impact on Output Performance](https://mlsanity.com/paper/2605.19645/) (2026)
- [FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference](https://mlsanity.com/paper/2608.24945/) (2026)
- [FlexPosit: Tunable Fractional Precision for LLM Inference Accelerators](https://mlsanity.com/paper/2609.04724/) (2026)
- [Analysis of Quantized and Efficiently Adapted Protein Language Models](https://mlsanity.com/paper/2610.00665/) (2026)
- [Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2](https://mlsanity.com/paper/2610.01889/) (2026)

## Abstract (arXiv)

Quantization has emerged as a promising technique for improving the memory and computational efficiency of large language models (LLMs). Though the trade-off between performance and efficiency is well-known, there is still much to be learned about the relationship between quantization and LLM performance. To shed light on this relationship, we propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We call this approach "the lens of perturbation". Using this lens, we conduct experiments with various artificial perturbations to explore their impact on LLM performance. Our findings reveal several connections between the properties of perturbations and LLM performance, providing insights into the failure cases of uniform quantization and suggesting potential solutions to improve the robustness of LLM quantization. To demonstrate the significance of our findings, we implement a simple non-uniform quantization approach based on our insights. Our experiments show that this approach achieves minimal performance degradation on both 4-bit weight quantization and 8-bit quantization for weights and activations. These results validate the correctness of our approach and highlight its potential to improve the efficiency of LLMs without sacrificing performance.