mlsanityData to 14 Oct 2026

StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

arXiv 2607.08993 ↗First posted 9 Jul 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

As large language models (LLMs) scale, their memory and computation demands have grown substantially, making weight-only quantization a widely adopted technique for reducing model size with minimal accuracy loss. However, on current GPUs, CUDA-core-based dequantization introduces substantial instruction overhead, on-chip traffic, and pipeline stalls, making it a major bottleneck for high-throughput, cloud-scale LLM serving. To address these limitations, we propose StreamDQ, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference. StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM) and performs inline dequantization on standard memory loads. A lightweight sideband tag on each memory read request selects the dequantization mode while preserving conventional load semantics. By relocating dequantization to the memory side, StreamDQ eliminates GPU-side CUDA-core-based dequantization, thereby reducing on-chip traffic on the GPU and avoiding extra HBM write-back and reload of dequantized weights at large batch sizes. Our evaluation shows that StreamDQ achieves up to 7.08× speedup and 90.23% lower energy for mixed-precision GEMM, with only 0.127,mm2 area and 0.355,W power overhead per DQB in a 12,nm CMOS process. For end-to-end LLM inference, StreamDQ reduces latency by up to 54.68% and improves decode throughput by up to 2.20×.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 61 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

21 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference25 Nov
  2. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format24 Nov
  3. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  4. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference15 Feb
  5. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  9. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  10. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  11. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
  12. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
4 more
  1. 2019RecNMP: Accelerating Personalized Recommendation with Near-Memory Processingcited by 49
  2. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 18
  3. 2020Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Acceleratorscited by 7
  4. 2016Near-Memory Address Translationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
  2. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  3. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
  4. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  5. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization
  6. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
  7. 2025Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms
  8. 2023Efficient LLM Inference on CPUs