mlsanityData to 7 Oct 2026

FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design

arXiv 2508.04405 ↗First posted 6 Aug 2025cs.LGcited by 2

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large Language Models (LLMs) demonstrate exceptional performance but entail significant memory and computational costs, restricting their practical deployment. While existing INT4/INT8 quantization reduces these costs, they often degrade accuracy or lack optimal efficiency. INT6 quantization offers a superior trade-off between model accuracy and inference efficiency, but lacks hardware support in modern GPUs, forcing emulation via higher-precision arithmetic units that limit acceleration. In this paper, we propose FlexQ, a novel post-training INT6 quantization framework combining algorithmic innovation with system-level optimizations. FlexQ employs uniform 6-bit weight quantization across all layers, with adaptive retention of 8-bit activations in layers identified through layer-wise sensitivity analysis. To maximize hardware efficiency, we develop a specialized high-performance GPU kernel supporting matrix multiplication for W6A6 and W6A8 representations via Binary Tensor Core (BTC) equivalents, effectively bypassing the lack of native INT6 tensor cores. Evaluations on LLaMA family models show FlexQ maintains near-FP16 accuracy, with perplexity increases of no more than 0.1 on WikiText2. The proposed kernel achieves an average 1.39× speedup over ABQ-LLM on LLaMA-2-70B linear layers. End-to-end, FlexQ delivers 1.33× inference acceleration and 1.21× memory savings over SmoothQuant. Code is released at https://github.com/FlyFoxPlayer/FlexQ.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 46 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

32 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 29

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
17 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  3. 2021A White Paper on Neural Network Quantizationcited by 453
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  6. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  7. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  8. 2023Microscaling Data Formats for Deep Learningcited by 132
  9. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  10. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70
  11. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  12. 2021QGTC: Accelerating Quantized Graph Neural Networks via GPU Tensor Corecited by 18
  13. 2019Dissecting the NVidia Turing T4 GPU via Microbenchmarkingcited by 15
  14. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inferencecited by 15
  15. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  16. 2021APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Corescited by 12
  17. 2024COMET: Towards Partical W4A4KV4 LLMs Servingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Mapping the Schedule x Bit-Width Boundary in Sub-100M Quantisation-Aware Trainingcited by 2
  2. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
  2. 2026AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
  3. 2025FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration
  4. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  5. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  6. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  7. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
  8. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization