mlsanityData to 14 Oct 2026

DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs

arXiv 2410.12187 ↗First posted 16 Oct 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel in various tasks but face deployment challenges due to hardware constraints. We propose density-aware post-training weight-only quantization (DAQ), which has two stages: 1) density-centric alignment, which identifies the center of high-density weights and centers the dynamic range on this point to align high-density weight regions with floating-point high-precision regions; 2) learnable dynamic range adjustment, which adjusts the dynamic range by optimizing quantization parameters (i.e., scale and zero-point) based on the impact of weights on the model output. Experiments on LLaMA and LLaMA-2 show that DAQ consistently outperforms the best baseline method, reducing perplexity loss by an average of 22.8% on LLaMA and 19.6% on LLaMA-2. Our code is available at https://github.com/LuoYingSong/DAQ.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

18 of 31 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  10. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
6 more
  1. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  2. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  3. 2019Loss Aware Post-training Quantizationcited by 97
  4. 2022FP8 Quantization: The Power of the Exponentcited by 88
  5. 2023NF4 Isn't Information Theoretically Optimal (and that's Good)cited by 11
  6. 2024QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs
  2. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
  3. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models
  4. 2026ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
  5. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization
  6. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
  7. 2025Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
  8. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization