# GWQ: Gradient-Aware Weight Quantization for Large Language Models

arXiv: https://arxiv.org/abs/2411.00850 · HTML: https://mlsanity.com/paper/2411.00850/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2024.

## Built on by (most cited first, 8 of 8)

- [Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation](https://mlsanity.com/paper/2601.10880/) (2026)
- [3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting](https://mlsanity.com/paper/2412.01583/) (2024)
- [TR-DQ: Time-Rotation Diffusion Quantization](https://mlsanity.com/paper/2503.06564/) (2025)
- [In-Context Meta LoRA Generation](https://mlsanity.com/paper/2501.17635/) (2025)
- [Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs](https://mlsanity.com/paper/2506.09104/) (2025)
- [Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression](https://mlsanity.com/paper/2504.07389/) (2025)
- [GM-MoE: Low-Light Enhancement with Gated-Mechanism Mixture-of-Experts](https://mlsanity.com/paper/2503.07417/) (2025)
- [Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction](https://mlsanity.com/paper/2609.39960/) (2026)

## Abstract (arXiv)

Large language models (LLMs) show impressive performance in solving complex language tasks. However, its large number of parameters presents significant challenges for the deployment. So, compressing LLMs to low bits can enable to deploy on resource-constrained devices. To address this problem, we propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization that leverages gradients to localize outliers, requiring only a minimal amount of calibration data for outlier detection. GWQ retains the top 1% outliers preferentially at FP16 precision, while the remaining non-outlier weights are stored in a low-bit. We widely evaluate GWQ on different task include language modeling, grounding detection, massive multitask language understanding and vision-language question and answering. Results show that models quantified by GWQ performs better than other quantization method. During quantization process, GWQ only need one calibration set to realize effective quant. Also, GWQ achieves 1.2x inference speedup in comparison to the original model and effectively reduces the inference memory.