mlsanityData to 7 Oct 2026

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models

arXiv 2402.14866 ↗First posted 21 Feb 2024cs.LG cs.AI cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have greatly advanced the natural language processing paradigm. However, the high computational load and huge model sizes pose a grand challenge for deployment on edge devices. To this end, we propose APTQ (Attention-aware Post-Training Mixed-Precision Quantization) for LLMs, which considers not only the second-order information of each layer's weights, but also, for the first time, the nonlinear effect of attention outputs on the entire model. We leverage the Hessian trace as a sensitivity metric for mixed-precision quantization, ensuring an informed precision reduction that retains model performance. Experiments show APTQ surpasses previous quantization methods, achieving an average of 4 bit width a 5.22 perplexity nearly equivalent to full precision in the C4 dataset. In addition, APTQ attains state-of-the-art zero-shot accuracy of 68.24% and 70.48% at an average bitwidth of 3.8 in LLaMa-7B and LLaMa-13B, respectively, demonstrating its effectiveness to produce high-quality quantized LLMs.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 9 of its 21 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

9 of 21 references.

References · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  6. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  7. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  8. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  9. 2019Progressive DARTS: Bridging the Optimization Gap for NAS in the Wildcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 10.

33 papers.

Sort:

By area:

  1. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  2. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  3. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  4. 2024A Comprehensive Study on Quantization Techniques for Large Language Modelscited by 34
  5. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  6. 2025Block Rotation is All You Need for MXFP4 Quantizationcited by 18
  7. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  8. 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 8
  9. 2025SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Modelscited by 7
  10. 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantizationcited by 7
  11. 2025RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithmcited by 5
  12. 2025Radio: Rate-Distortion Optimization for Large Language Model Compressioncited by 4
21 more
  1. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  2. 2025Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retrainingcited by 3
  3. 2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
  4. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMscited by 3
  5. 2026OneComp: One-Line Revolution for Generative AI Model Compressioncited by 3
  6. 2024Foundations of Large Language Model Compression -- Part 1: Weight Quantizationcited by 2
  7. 2024Continuous Approximations for Improving Quantization Aware Training of LLMscited by 2
  8. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2
  9. 2025FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMscited by 2
  10. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  11. 2026dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formatscited by 2
  12. 2024LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Modelscited by 1
  13. 2025Pack-PTQ: Advancing Post-training Quantization of Neural Networks by Pack-wise Reconstructioncited by 1
  14. 2025Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Methodcited by 1
  15. 2025KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computingcited by 1
  16. 2026Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantizationcited by 1
  17. 2025Irrational Complex Rotations Empower Low-bit Optimizerscited by 0
  18. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Trainingcited by 0
  19. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0
  20. 2026Component Type, Not Reconstruction Error, Predicts Attention Quantization Sensitivitycited by 0
  21. 2026From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformerscited by 0