mlsanityData to 7 Oct 2026

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

arXiv 2610.07457 ↗First posted 5 Oct 2026cs.LG cs.AI cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units. This precision-boundary mismatch limits the translation of compression into practical acceleration. We introduce AlignQuant, a post-training quantization method that uses GPU-compatible two-dimensional weight tiles as the common unit of precision allocation, compact storage, and execution. This shared partition lets precision follow sensitivity within output channels. Joint prefill/decode calibration scores precision reductions using projection-output perturbations weighted by language-model loss gradients under quantized activations. Phase-normalized scores prioritize higher precision for tiles important to either phase under a model-wide weight-storage budget. Each tile stores one selected representation, while phase-specialized kernels reuse the packed model and expand lower-bit weights for INT8 computation with 8-bit activations. Across four LLMs spanning 3B to 14B parameters, AlignQuant achieves up to 2.50× generation speedup over BF16 while preserving model quality. Evaluations further cover three GPUs and contexts up to 64K tokens. These results show that local precision flexibility and regular GPU execution can coexist through a shared tile unit. The implementation is available at https://github.com/HanzhiZhang-Ulrica/AlignQuant.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 41 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

38 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference20 Jul
  2. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving1 Sep
  3. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference2 Jan
  4. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inference17 Oct
  5. 2024SpinQuant: LLM quantization with learned rotations26 May
  6. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 30

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  12. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
18 more
  1. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  2. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  3. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  4. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 64
  5. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  6. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  7. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  8. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  9. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 15
  10. 2024Channel-Wise Mixed-Precision Quantization for Large Language Modelscited by 11
  11. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
  12. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inferencecited by 9
  13. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 9
  14. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMscited by 8
  15. 2024Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inferencecited by 5
  16. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMscited by 3
  17. 2025FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMscited by 2
  18. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  2. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospects
  3. 2025FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design
  4. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
  5. 2026ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
  6. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
  7. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs
  8. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models