Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
Not on the map. Why?
Abstract (arXiv)
The ever-increasing sizes of large language models necessitate distributed solutions for fast inference that exploit multi-dimensional parallelism, where computational loads are split across various accelerators such as GPU clusters. However, this approach often introduces significant communication overhead, especially on devices with limited bandwidth. In this paper, we introduce Flash Communication, a novel low-bit compression technique designed to alleviate the tensor-parallelism communication bottleneck during inference. Our method substantially boosts intra-node communication speed by more than 3x and reduces the time-to-first-token by 2x, with nearly no sacrifice in model accuracy. Extensive experiments on various up-to-date LLMs demonstrate the effectiveness of our approach.
Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2024.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 6; 2026: 5.
11 papers.
- 2025Tensor-Parallelism with Partially Synchronized Activationscited by 4
- 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inferencecited by 3
- 2025FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concerncited by 2
- 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloadscited by 2
- 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communicationcited by 1
- 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
- 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
- 2026A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Networkcited by 0
- 2026CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Trainingcited by 0
- 2026Affinity-Aware Sharding for Delayed Tensor Parallelismcited by 0
- 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inferencecited by 0
- 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference9 Oct 2026
- 2026Affinity-Aware Sharding for Delayed Tensor Parallelism12 Sep 2026
- 2026CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training27 Apr 2026
- 2026A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network30 Mar 2026
- 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads29 Jan 2026
- 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks4 Nov 2025
- 2025FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern30 Aug 2025
- 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference26 Aug 2025
- 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference21 Aug 2025
- 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication4 Aug 2025
- 2025Tensor-Parallelism with Partially Synchronized Activations24 Jun 2025
Similar, not cited
Close in wording, no citation either way
- 2024Communication Compression for Tensor Parallel LLM Inference
- 2024Towards Low-bit Communication for Tensor Parallel LLM Inference
- 2023FlashDecoding++: Faster Large Language Model Inference on GPUs
- 2025Characterizing Communication Patterns in Distributed Large Language Model Inference
- 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
- 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
- 2026Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
- 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads