mlsanityData to 14 Oct 2026

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

arXiv 2406.09904 ↗First posted 14 Jun 2024cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference. W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation. To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations. QQQ employs adaptive smoothing and Hessian-based compensation, significantly enhancing the performance of quantized models without extensive training. Furthermore, we meticulously engineer W4A8 GEMM kernels to increase inference speed. Our specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve impressive speed increases of 3.67× and 3.29 × over FP16 GEMM. Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 ×, 2.10×, and 1.25× compared to FP16, W8A8, and W4A16, respectively.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

18 of 26 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
6 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  6. 2023A Speed Odyssey for Deployable Quantization of LLMscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 59
  2. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
  3. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  4. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 16
  5. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  6. 2026FAAR: Format-Aware Adaptive Rounding for NVFP4cited by 4
  7. 2025Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Designcited by 3
  8. 2026W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUscited by 3
  9. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  10. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inferencecited by 1
  11. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Modelscited by 1
  12. 2026APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancingcited by 1
4 more
  1. 2026Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0cited by 0
  2. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Designcited by 0
  3. 2026Predicting Quantization Price for Selecting PTQ Configurations Before Deploymentcited by 0
  4. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  3. 2025BAQ: Efficient Bit Allocation Quantization for Large Language Models
  4. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
  5. 2025SiLQ: Simple Large Language Model Quantization-Aware Training
  6. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
  7. 2025An Empirical Study of Qwen3 Quantization
  8. 2024Post Training Quantization of Large Language Models with Microscaling Formats