mlsanityData to 14 Oct 2026

Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads

arXiv 2105.05720 ↗First posted 12 May 2021cs.DC cs.LG cs.PLcited by 40

Not on the map. Why?

Abstract (arXiv)

Recent trend towards increasing large machine learning models require both training and inference tasks to be distributed. Considering the huge cost of training these models, it is imperative to unlock optimizations in computation and communication to obtain best performance. However, current logical separation between computation and communication kernels in deep learning frameworks misses the optimization opportunities across such barrier. Breaking this abstraction with a holistic consideration can provide many optimizations to provide performance improvements in distributed workloads. Manually applying these optimizations needs modifications in underlying computation and communication libraries for each scenario, which is time consuming and error-prone. Therefore, we present CoCoNeT, with a DSL to express a program with both computation and communication. CoCoNeT contains several machine learning aware transformations to optimize a program and a compiler to generate high performance kernels. Providing both computation and communication as first class constructs allows users to work on a high-level abstraction and apply powerful optimizations, such as fusion or overlapping of communication and computation. CoCoNeT enables us to optimize data-, model-and pipeline-parallel workloads in large language models with only a few lines of code. Experiments show CoCoNeT significantly outperforms state-of-the-art distributed machine learning implementations.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 16 of its 54 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

16 of 54 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  11. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  12. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
4 more
  1. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  2. 2018TVM: An Automated End-to-End Optimizing Compiler for Deep Learningcited by 62
  3. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  4. 2017Loop Tiling in Large-Scale Stencil Codes at Run-time with OPScited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 10; 2025: 15; 2026: 10.

40 papers.

Sort:

By area:

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  2. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  3. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 49
  4. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  5. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  6. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  7. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  8. 2022On Optimizing the Communication of Model Parallelismcited by 19
  9. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  10. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 16
  11. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  12. 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 14
28 more
  1. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 14
  2. 2024Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfacescited by 11
  3. 2025MSCCL++: Rethinking GPU Communication Abstractions for AI Inferencecited by 10
  4. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 9
  5. 2025Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlappingcited by 9
  6. 2023A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUscited by 6
  7. 2025An Inquiry into Datacenter TCO for LLM Inference with FP8cited by 5
  8. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 5
  9. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 5
  10. 2022MPC-Pipe: an Efficient Pipeline Scheme for Secure Multi-party Machine Learning Inferencecited by 4
  11. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  12. 2025Tensor-Parallelism with Partially Synchronized Activationscited by 4
  13. 2024ISO: Overlap of Computation and Communication within Seqenence For LLM Inferencecited by 3
  14. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  15. 2026Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Trainingcited by 3
  16. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  17. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Trainingcited by 2
  18. 2026CUCo: An Agentic Framework for Compute and Communication Co-designcited by 2
  19. 2025Efficient AllReduce with Stragglerscited by 1
  20. 2025Design Space Exploration of DMA based Finer-Grain Compute Communication Overlapcited by 1
  21. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Expertscited by 1
  22. 2025ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniquescited by 0
  23. 2026CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Trainingcited by 0
  24. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUscited by 0
  25. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systemscited by 0
  26. 2026Resource-aware Computation-Communication Overlap for multi-GPU ML Workloadscited by 0
  27. 2026X-Stage: Modeling Post-Issue Backpressure in GPU Communication--Computation Fusioncited by 0
  28. 2026mKernel: Fast Multi-GPU, Multi-Node Fused Kernelscited by 0