mlsanityData to 7 Oct 2026

MCR-DL: Mix-and-Match Communication Runtime for Deep Learning

arXiv 2303.08374 ↗First posted 15 Mar 2023cs.DC cs.LGcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

In recent years, the training requirements of many state-of-the-art Deep Learning (DL) models have scaled beyond the compute and memory capabilities of a single processor, and necessitated distribution among processors. Training such massive models necessitates advanced parallelism strategies to maintain efficiency. However, such distributed DL parallelism strategies require a varied mixture of collective and point-to-point communication operations across a broad range of message sizes and scales. Examples of models using advanced parallelism strategies include Deep Learning Recommendation Models (DLRM) and Mixture-of-Experts (MoE). Communication libraries' performance varies wildly across different communication operations, scales, and message sizes. We propose MCR-DL: an extensible DL communication framework that supports all point-to-point and collective operations while enabling users to dynamically mix-and-match communication backends for a given operation without deadlocks. MCR-DL also comes packaged with a tuning suite for dynamically selecting the best communication backend for a given input tensor. We select DeepSpeed-MoE and DLRM as candidate DL models and demonstrate a 31% improvement in DS-MoE throughput on 256 V100 GPUs on the Lassen HPC system. Further, we achieve a 20% throughput improvement in a dense Megatron-DeepSpeed model and a 25% throughput improvement in DLRM on 32 A100 GPUs with the Theta-GPU HPC system.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 16 of its 44 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

16 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  9. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  10. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  11. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 87
  12. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
3 more
  1. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  2. 2020Synthesizing Optimal Collective Algorithmscited by 30
  3. 2022MPIX Stream: An Explicit Solution to Hybrid MPI+X Programmingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  2. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  3. 2024Accelerating Large Language Model Training with Hybrid GPU-based Compressioncited by 2
  4. 2026Concurrent Split Learning Through Stable Client Clusteringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
  2. 2025Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach
  3. 2024MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimization
  4. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  5. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operations
  6. 2023DISCO: Distributed Inference with Sparse Communications
  7. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
  8. 2025BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference