mlsanityData to 14 Oct 2026

FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion

arXiv 2406.06858 ↗First posted 11 Jun 2024cs.LG cs.DCcited by 57

Not on the map. Why?

Abstract (arXiv)

Large deep learning models have demonstrated strong ability to solve many tasks across a wide range of applications. Those large models typically require training and inference to be distributed. Tensor parallelism is a common technique partitioning computation of an operation or layer across devices to overcome the memory capacity limitation of a single processor, and/or to accelerate computation to meet a certain latency requirement. However, this kind of parallelism introduces additional communication that might contribute a significant portion of overall runtime. Thus limits scalability of this technique within a group of devices with high speed interconnects, such as GPUs with NVLinks in a node. This paper proposes a novel method, Flux, to significantly hide communication latencies with dependent computations for GPUs. Flux over-decomposes communication and computation operations into much finer-grained operations and further fuses them into a larger kernel to effectively hide communication without compromising kernel efficiency. Flux can potentially overlap up to 96% of communication given a fused kernel. Overall, it can achieve up to 1.24x speedups for training over Megatron-LM on a cluster of 128 GPUs with various GPU generations and interconnects, and up to 1.66x and 1.30x speedups for prefill and decoding inference over vLLM on a cluster with 8 GPUs with various GPU generations and interconnects.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

29 of 49 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2023BloombergGPT: A Large Language Model for Financecited by 939
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
17 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  3. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  4. 2022Efficiently Scaling Transformer Inferencecited by 539
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  7. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  8. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  9. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  10. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  11. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  12. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  13. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  14. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  15. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  16. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  17. 2023Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPUcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 25; 2026: 26.

57 papers.

Sort:

By area:

  1. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  2. 2025Is Artificial Intelligence Generated Image Detection a Solved Problem?cited by 51
  3. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 49
  4. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  5. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  6. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  7. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  8. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  9. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 19
  10. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  11. 2025Robust LLM Training Infrastructure at ByteDancecited by 16
  12. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
38 more
  1. 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 14
  2. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 10
  3. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 9
  4. 2025Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlappingcited by 9
  5. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Servingcited by 9
  6. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformerscited by 6
  7. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 5
  8. 2025Collective Communication for 100k+ GPUscited by 4
  9. 2024ISO: Overlap of Computation and Communication within Seqenence For LLM Inferencecited by 3
  10. 2025Leveraging Large Language Models For Scalable Vector Graphics Processing: A Reviewcited by 3
  11. 2025Constructing a 3D Scene from a Single Imagecited by 3
  12. 2025Self-Evolving 3D Scene Generation from a Single Imagecited by 3
  13. 2026Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Trainingcited by 3
  14. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  15. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2
  16. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUscited by 2
  17. 2026CUCo: An Agentic Framework for Compute and Communication Co-designcited by 2
  18. 2026UCCL-Zip: Lossless Compression Supercharged GPU Communicationcited by 2
  19. 2025Could AI Trace and Explain the Origins of AI-Generated Images and Text?cited by 1
  20. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuningcited by 1
  21. 2025Design Space Exploration of DMA based Finer-Grain Compute Communication Overlapcited by 1
  22. 2026Exploiting the Final Component of Generator Architectures for AI-Generated Image Detectioncited by 1
  23. 2026Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronizationcited by 1
  24. 2026Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Trainingcited by 1
  25. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernelscited by 1
  26. 2024Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMMcited by 0
  27. 2024Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linkingcited by 0
  28. 2025Animalbooth: multimodal feature enhancement for animal subject personalizationcited by 0
  29. 2025AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Trainingcited by 0
  30. 2025Iris: First-Class Multi-GPU Programming Experience in Tritoncited by 0
  31. 2026CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Trainingcited by 0
  32. 2026DynaFlow: Transparent and Flexible Intra-Device Parallelism via Programmable Operator Schedulingcited by 0
  33. 2026HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUscited by 0
  34. 2026Resource-aware Computation-Communication Overlap for multi-GPU ML Workloadscited by 0
  35. 2026Piper: A Programmable Distributed Training Systemcited by 0
  36. 2026CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systemscited by 0
  37. 2026SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalizationcited by 0
  38. 2026Analytical Resource Management for Fine-grained MoE Computation-Communication Overlapcited by 0

50 most cited of 57 All 57 (JSON)