mlsanityData to 14 Oct 2026

HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference

arXiv 2609.00450 ↗First posted 31 Aug 2026cs.LG cs.AI cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Block Quantization (BQ) enables efficient LLM inference by quantizing both weights and activations, but its design space remains underexplored. Through hardware-accuracy design space exploration, we identify block size as a key trade-off: larger blocks improve hardware efficiency by amortizing dequantization and accumulation costs, but degrade accuracy. Motivated by this insight, we propose Hierarchical Block Quantization (HBQ), which combines large blocks with low-overhead significand (SIG) scaling for second-level quantization. SIG scaling effectively compensates for large-block quantization errors while accounting for distinct weight and activation distributions. HBQ-A achieves W4A16-level accuracy with W4A5 and lower area than NVFP4, while HBQ-E further reduces hardware cost by 17% while outperforming existing BQ methods in accuracy. We implement HBQ for weights, activations, and KV cache in a 28nm ASIC accelerator and introduce partial-sum BQ to reduce EMA energy. At comparable accuracy, HBQ achieves 2.3x/4.6x higher area/energy efficiency than state-of-the-art weight-only quantization and 1.6-3.3x lower system energy with 1.5-3x speedup over prior BQ methods. Our implementation is publicly available at: https://github.com/SeoLabCornell/HBQ.git.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 63 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

46 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving16 Oct
  2. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference2 Jan
  3. 2024SpinQuant: LLM quantization with learned rotations26 May
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 40

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2024Mixtral of Expertscited by 1,823
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
28 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  2. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  3. 2022FP8 Formats for Deep Learningcited by 196
  4. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  5. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  6. 2023Microscaling Data Formats for Deep Learningcited by 152
  7. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  8. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  9. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  10. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  11. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
  12. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  13. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  14. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  15. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  16. 2018Training DNNs with Hybrid Block Floating Pointcited by 40
  17. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  18. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  19. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 19
  20. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcited by 17
  21. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 16
  22. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 15
  23. 2024Post Training Quantization of Large Language Models with Microscaling Formatscited by 12
  24. 2025KurTail : Kurtosis-based LLM Quantizationcited by 12
  25. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 8
  26. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 7
  27. 2024OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Modelscited by 4
  28. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
  2. 2024Accurate Block Quantization in LLMs with Outliers
  3. 2026Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference
  4. 2025HALO: Hardware-aware quantization with low critical-path-delay weights for LLM acceleration
  5. 2023Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
  6. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs
  7. 2026StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration
  8. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs