# APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models

arXiv: https://arxiv.org/abs/2402.14866 · HTML: https://mlsanity.com/paper/2402.14866/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [LLM-FP4: 4-Bit Floating-Point Quantized Transformers](https://mlsanity.com/paper/2310.16836/) (2023)
- [Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild](https://mlsanity.com/paper/1912.10952/) (2019)

## Built on by (most cited first, 30 of 34)

- [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2405.14917/) (2024)
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [A Comprehensive Study on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2411.02530/) (2024)
- [ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals](https://mlsanity.com/paper/2412.14363/) (2024)
- [Block Rotation is All You Need for MXFP4 Quantization](https://mlsanity.com/paper/2511.04214/) (2025)
- [Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis](https://mlsanity.com/paper/2502.13178/) (2025)
- [ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms](https://mlsanity.com/paper/2509.09679/) (2025)
- [SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models](https://mlsanity.com/paper/2508.03351/) (2025)
- [TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization](https://mlsanity.com/paper/2606.13054/) (2026)
- [RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm](https://mlsanity.com/paper/2504.03717/) (2025)
- [Radio: Rate-Distortion Optimization for Large Language Model Compression](https://mlsanity.com/paper/2505.03031/) (2025)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining](https://mlsanity.com/paper/2504.13932/) (2025)
- [You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations](https://mlsanity.com/paper/2511.06516/) (2025)
- [ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs](https://mlsanity.com/paper/2602.17698/) (2026)
- [OneComp: One-Line Revolution for Generative AI Model Compression](https://mlsanity.com/paper/2603.28845/) (2026)
- [dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats](https://mlsanity.com/paper/2606.04115/) (2026)
- [Foundations of Large Language Model Compression -- Part 1: Weight Quantization](https://mlsanity.com/paper/2409.02026/) (2024)
- [Continuous Approximations for Improving Quantization Aware Training of LLMs](https://mlsanity.com/paper/2410.10849/) (2024)
- [HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecture](https://mlsanity.com/paper/2502.19747/) (2025)
- [SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training](https://mlsanity.com/paper/2602.01410/) (2026)
- [EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation](https://mlsanity.com/paper/2605.04062/) (2026)
- [LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models](https://mlsanity.com/paper/2408.10631/) (2024)
- [FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs](https://mlsanity.com/paper/2504.19746/) (2025)
- [Pack-PTQ: Advancing Post-training Quantization of Neural Networks by Pack-wise Reconstruction](https://mlsanity.com/paper/2505.00259/) (2025)
- [Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method](https://mlsanity.com/paper/2507.18073/) (2025)
- [KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing](https://mlsanity.com/paper/2512.03608/) (2025)
- [Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization](https://mlsanity.com/paper/2607.00908/) (2026)
- [Irrational Complex Rotations Empower Low-bit Optimizers](https://mlsanity.com/paper/2501.12896/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have greatly advanced the natural language processing paradigm. However, the high computational load and huge model sizes pose a grand challenge for deployment on edge devices. To this end, we propose APTQ (Attention-aware Post-Training Mixed-Precision Quantization) for LLMs, which considers not only the second-order information of each layer's weights, but also, for the first time, the nonlinear effect of attention outputs on the entire model. We leverage the Hessian trace as a sensitivity metric for mixed-precision quantization, ensuring an informed precision reduction that retains model performance. Experiments show APTQ surpasses previous quantization methods, achieving an average of 4 bit width a 5.22 perplexity nearly equivalent to full precision in the C4 dataset. In addition, APTQ attains state-of-the-art zero-shot accuracy of 68.24% and 70.48% at an average bitwidth of 3.8 in LLaMa-7B and LLaMa-13B, respectively, demonstrating its effectiveness to produce high-quality quantized LLMs.