# Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference

arXiv: https://arxiv.org/abs/2412.04964 · HTML: https://mlsanity.com/paper/2412.04964/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2024.

## Built on by (most cited first, 11 of 11)

- [Tensor-Parallelism with Partially Synchronized Activations](https://mlsanity.com/paper/2506.19645/) (2025)
- [HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference](https://mlsanity.com/paper/2508.19373/) (2025)
- [FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern](https://mlsanity.com/paper/2510.15882/) (2025)
- [Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads](https://mlsanity.com/paper/2601.21351/) (2026)
- [FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication](https://mlsanity.com/paper/2508.03760/) (2025)
- [TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference](https://mlsanity.com/paper/2508.15881/) (2025)
- [Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks](https://mlsanity.com/paper/2511.02647/) (2025)
- [A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network](https://mlsanity.com/paper/2603.28239/) (2026)
- [CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training](https://mlsanity.com/paper/2604.24013/) (2026)
- [Affinity-Aware Sharding for Delayed Tensor Parallelism](https://mlsanity.com/paper/2609.13846/) (2026)
- [Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference](https://mlsanity.com/paper/2610.14277/) (2026)

## Abstract (arXiv)

The ever-increasing sizes of large language models necessitate distributed solutions for fast inference that exploit multi-dimensional parallelism, where computational loads are split across various accelerators such as GPU clusters. However, this approach often introduces significant communication overhead, especially on devices with limited bandwidth. In this paper, we introduce Flash Communication, a novel low-bit compression technique designed to alleviate the tensor-parallelism communication bottleneck during inference. Our method substantially boosts intra-node communication speed by more than 3x and reduces the time-to-first-token by 2x, with nearly no sacrifice in model accuracy. Extensive experiments on various up-to-date LLMs demonstrate the effectiveness of our approach.