# Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap arXiv: https://arxiv.org/abs/2512.10236 ยท HTML: https://mlsanity.com/paper/2512.10236/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 67 references that are in the map. - First posted on arXiv 11 Dec 2025. ## Builds on (most cited first, 30 of 32) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022) - [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024) - [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020) - [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023) - [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024) - [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021) - [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023) - [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022) - [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022) - [FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion](https://mlsanity.com/paper/2406.06858/) (2024) - [Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts](https://mlsanity.com/paper/2502.19811/) (2025) - [A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training](https://mlsanity.com/paper/2303.06318/) (2023) - [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021) - [Petuum: A New Platform for Distributed Machine Learning on Big Data](https://mlsanity.com/paper/1312.7651/) (2013) - [Context Parallelism for Scalable Million-Token Inference](https://mlsanity.com/paper/2411.01783/) (2024) - [Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms](https://mlsanity.com/paper/2007.00156/) (2020) - [Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler](https://mlsanity.com/paper/2504.19442/) (2025) - [TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives](https://mlsanity.com/paper/2503.20313/) (2025) - [Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU](https://mlsanity.com/paper/2301.03598/) (2023) - [Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms](https://mlsanity.com/paper/2507.04786/) (2025) - [Optimizing Distributed ML Communication with Fused Computation-Collective Operations](https://mlsanity.com/paper/2305.06942/) (2023) - [Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering](https://mlsanity.com/paper/2504.19519/) (2025) - [Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping](https://mlsanity.com/paper/2409.15241/) (2024) - [Power Stabilization for AI Training Datacenters](https://mlsanity.com/paper/2508.14318/) (2025) - [GPU-Initiated Networking for NCCL](https://mlsanity.com/paper/2511.15076/) (2025) ## Built on by (most cited first, 1 of 1) - [Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs](https://mlsanity.com/paper/2609.40093/) (2026) ## Abstract (arXiv) Modern ML workloads demand distributing training and inference across multiple GPUs. However, these parallelization techniques often suffer from exposed critical-path communication, leaving a potential 1.7x speedup on the table through compute-communication overlap. Prior overlapping methods harness the fact that ML model state and inputs are already sharded into the number of GPUs, and overlap the compute and communication at shard granularity. However, such coarse-grained overlap suffers from limited network topology support, and suboptimal dataflows. In this work, we instead make a case for finer-grain compute-communication overlap which we term FiCCO. FiCCO operates one level deeper than traditional sharding, and unlocks overlap for a wider set of network topologies and enables finer-grain dataflow. We show that FiCCO opens up a wider design space of execution schedules than possible at shard-level alone. To walk the design space of schedules, we study and characterize the performance inefficiencies on doing overlap and overlay the schedules with the associated inefficiency signatures. Our characterization reveals decomposition and contention based slowdowns to be the major performance limiters, and we correlate the slowdown factors with the static compute/communication operator sizes. This helps us design heuristics (that frameworks and runtimes can harness) to select bespoke FiCCO schedules based on the nature of underlying ML operations. Finally, to further minimize contention inefficiencies inherent with operation overlap, we offload communication to GPU DMA engines. We evaluate several scenarios from realistic ML deployments and demonstrate that our proposed heuristics driven bespoke schedules deliver up to 1.6x speedup. Further, our heuristics provide accurate guidance to pick the optimal schedule in 81% of unseen scenarios.