mlsanityData to 14 Oct 2026

FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication

arXiv 2508.03760 ↗First posted 4 Aug 2025cs.DC cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashCommunication V2, a novel communication paradigm enabling efficient cross-GPU transmission at arbitrary bit widths. Its core innovations lie in the proposed bit splitting and spike reserving techniques, which address the challenges of low-bit quantization. Bit splitting decomposes irregular bit widths into basic units, ensuring compatibility with hardware capabilities and thus enabling transmission at any bit width. Spike reserving, on the other hand, retains numerical outliers (i.e., minima and maxima) as floating-point numbers, which shrinks the dynamic numerical range and pushes the quantization limits to 2-bit with acceptable losses. FlashCommunication V2 significantly enhances the flexibility and resource utilization of communication systems. Through meticulous software-hardware co-design, it delivers robust performance and reduced overhead across both NVLink-based and PCIe-based architectures, achieving a maximum 3.2× speedup in AllReduce and 2× in All2All communication.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

20 of 28 references.

References · 20

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  9. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  12. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
8 more
  1. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  2. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  3. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  4. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  5. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  6. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  7. 2024Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inferencecited by 11
  8. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization
  2. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs
  3. 2024KVDirect: Distributed Disaggregated LLM Inference
  4. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗
  5. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  6. 2024FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs
  7. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  8. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training