mlsanityData to 14 Oct 2026

TP-Aware Dequantization

arXiv 2402.04925 ↗First posted 15 Jan 2024cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel method that reduces model inference latency during distributed deployment of Large Language Models (LLMs). Our contribution is an optimized inference deployment scheme that address the current limitations of state-of-the-art quantization kernels when used in conjunction with Tensor Parallel (TP). Our method preserves data locality in GPU memory access patterns and exploits a priori knowledge of TP to reduce global communication. We demonstrate an up to 1.81x speedup over existing methods for Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 1 of its 6 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

1 of 6 references.

References · 1

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast NF4 Dequantization Kernels for Large Language Model Inference
  2. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  3. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  4. 2026Scaling State-Space Models on Multiple GPUs with Tensor Parallelism
  5. 2024Communication Compression for Tensor Parallel LLM Inference
  6. 2026DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
  7. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  8. 2026Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization