# On Optimizing the Communication of Model Parallelism

arXiv: https://arxiv.org/abs/2211.05322 · HTML: https://mlsanity.com/paper/2211.05322/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [Beyond Data and Model Parallelism for Deep Neural Networks](https://mlsanity.com/paper/1807.05358/) (2018)
- [Mesh-TensorFlow: Deep Learning for Supercomputers](https://mlsanity.com/paper/1811.02084/) (2018)
- [Memory-Efficient Pipeline-Parallel DNN Training](https://mlsanity.com/paper/2006.09503/) (2020)
- [GSPMD: General and Scalable Parallelization for ML Computation Graphs](https://mlsanity.com/paper/2105.04663/) (2021)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [U-Net Transformer: Self and Cross Attention for Medical Image Segmentation](https://mlsanity.com/paper/2103.06104/) (2021)
- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [Supporting Very Large Models using Automatic Dataflow Graph Partitioning](https://mlsanity.com/paper/1807.08887/) (2018)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)
- [Blink: Fast and Generic Collectives for Distributed ML](https://mlsanity.com/paper/1910.04940/) (2019)
- [Synthesizing Optimal Collective Algorithms](https://mlsanity.com/paper/2008.08708/) (2020)
- [TicTac: Accelerating Distributed Deep Learning with Communication Scheduling](https://mlsanity.com/paper/1803.03288/) (2018)
- [Priority-based Parameter Propagation for Distributed DNN Training](https://mlsanity.com/paper/1905.03960/) (2019)
- [GC3: An Optimizing Compiler for GPU Collective Communication](https://mlsanity.com/paper/2201.11840/) (2022)
- [Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems](https://mlsanity.com/paper/2002.05814/) (2020)
- [Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning](https://mlsanity.com/paper/2110.10548/) (2021)
- [Memory-efficient array redistribution through portable collective communication](https://mlsanity.com/paper/2112.01075/) (2021)

## Built on by (most cited first, 19 of 19)

- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Optimizing RLHF Training for Large Language Models with Stage Fusion](https://mlsanity.com/paper/2409.13221/) (2024)
- [ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation](https://mlsanity.com/paper/2406.14088/) (2024)
- [Pipeline Parallelism with Controllable Memory](https://mlsanity.com/paper/2405.15362/) (2024)
- [Communication Compression for Tensor Parallel LLM Inference](https://mlsanity.com/paper/2411.09510/) (2024)
- [SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction](https://mlsanity.com/paper/2301.08984/) (2023)
- [Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment](https://mlsanity.com/paper/2312.03549/) (2023)
- [Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities](https://mlsanity.com/paper/2403.07585/) (2024)
- [Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search](https://mlsanity.com/paper/2311.15269/) (2023)
- [Ravnest: Decentralized Asynchronous Training on Heterogeneous Devices](https://mlsanity.com/paper/2401.01728/) (2024)
- [Layer-Parallel Training for Transformers](https://mlsanity.com/paper/2601.09026/) (2026)
- [Partitioned Neural Network Training via Synthetic Intermediate Labels](https://mlsanity.com/paper/2403.11204/) (2024)
- [Intra-DP: A High Performance Collaborative Inference System for Mobile Edge Computing](https://mlsanity.com/paper/2507.05829/) (2025)
- [CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks](https://mlsanity.com/paper/2509.19855/) (2025)
- [GaNDLF-Synth: A Framework to Democratize Generative AI for (Bio)Medical Imaging](https://mlsanity.com/paper/2410.00173/) (2024)
- [Collaborative Inference Acceleration with Non-Penetrative Tensor Partitioning](https://mlsanity.com/paper/2501.04489/) (2025)
- [Slicing Is All You Need: Towards A Universal One-Sided Algorithm for Distributed Matrix Multiplication](https://mlsanity.com/paper/2510.08874/) (2025)
- [Performance of Small Language Model Pretraining on FABRIC: An Empirical Study](https://mlsanity.com/paper/2602.02632/) (2026)

## Abstract (arXiv)

We study a novel and important communication pattern in large-scale model-parallel deep learning (DL), which we call cross-mesh resharding. This pattern emerges when the two paradigms of model parallelism - intra-operator and inter-operator parallelism - are combined to support large models on large clusters. In cross-mesh resharding, a sharded tensor needs to be sent from a source device mesh to a destination device mesh, on which the tensor may be distributed with the same or different layouts. We formalize this as a many-to-many multicast communication problem, and show that existing approaches either are sub-optimal or do not generalize to different network topologies or tensor layouts, which result from different model architectures and parallelism strategies. We then propose two contributions to address cross-mesh resharding: an efficient broadcast-based communication system, and an "overlapping-friendly" pipeline schedule. On microbenchmarks, our overall system outperforms existing ones by up to 10x across various tensor and mesh layouts. On end-to-end training of two large models, GPT-3 and U-Transformer, we improve throughput by 10% and 50%, respectively.