# TP-Aware Dequantization

arXiv: https://arxiv.org/abs/2402.04925 · HTML: https://mlsanity.com/paper/2402.04925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 6 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Builds on (most cited first, 1 of 1)

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)

## Abstract (arXiv)

In this paper, we present a novel method that reduces model inference latency during distributed deployment of Large Language Models (LLMs). Our contribution is an optimized inference deployment scheme that address the current limitations of state-of-the-art quantization kernels when used in conjunction with Tensor Parallel (TP). Our method preserves data locality in GPU memory access patterns and exploits a priori knowledge of TP to reduce global communication. We demonstrate an up to 1.81x speedup over existing methods for Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.