mlsanityData to 7 Oct 2026

Communication Compression for Tensor Parallel LLM Inference

arXiv 2411.09510 ↗First posted 14 Nov 2024cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

15 of 30 references.

References · 15

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2022Efficiently Scaling Transformer Inferencecited by 529
  10. 2024Large Language Models: A Surveycited by 506
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
3 more
  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  2. 2022On Optimizing the Communication of Model Parallelismcited by 19
  3. 2023Does compressing activations help model parallel training?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Modelscited by 4
  2. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 1
  3. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  4. 2026Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimizationcited by 0
  5. 2026Affinity-Aware Sharding for Delayed Tensor Parallelismcited by 0