mlsanityData to 14 Oct 2026

Towards Low-bit Communication for Tensor Parallel LLM Inference

arXiv 2411.07942 ↗First posted 12 Nov 2024cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Tensor parallelism provides an effective way to increase server large language model (LLM) inference efficiency despite adding an additional communication cost. However, as server LLMs continue to scale in size, they will need to be distributed across more devices, magnifying the communication cost. One way to approach this problem is with quantization, but current methods for LLMs tend to avoid quantizing the features that tensor parallelism needs to communicate. Taking advantage of consistent outliers in communicated features, we introduce a quantization method that reduces communicated values on average from 16 bits to 4.2 bits while preserving nearly all of the original performance. For instance, our method maintains around 98.0% and 99.5% of Gemma 2 27B's and Llama 2 13B's original performance, respectively, averaged across all tasks we evaluated on.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

23 of 26 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
11 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  6. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  7. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  8. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  9. 2020Distributed Inference with Sparse and Quantized Communicationcited by 3
  10. 2023DISCO: Distributed Inference with Sparse Communicationscited by 2
  11. 2013Distributed Inference with M-ary Quantized Data in the Presence of Byzantine Attackscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025Leveraging the true depth of LLMscited by 7
  2. 2025Scalable LLM Reasoning Acceleration with Low-rank Distillationcited by 7
  3. 2025SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Modelscited by 6
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2026TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Trainingcited by 2
  6. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  7. 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Communication Compression for Tensor Parallel LLM Inference
  2. 2024Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
  3. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  4. 2025ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decomposition
  5. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference
  6. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization
  7. 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
  8. 2024FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization