# MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms

arXiv: https://arxiv.org/abs/2209.06800 · HTML: https://mlsanity.com/paper/2209.06800/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 62 references that are in the map.
- First posted on arXiv 14 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [Memory-Efficient Pipeline-Parallel DNN Training](https://mlsanity.com/paper/2006.09503/) (2020)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks](https://mlsanity.com/paper/2004.13826/) (2020)
- [HyGCN: A GCN Accelerator with Hybrid Architecture](https://mlsanity.com/paper/2001.02514/) (2020)
- [Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect](https://mlsanity.com/paper/1903.04611/) (2019)
- [Learning Graph Representations with Embedding Propagation](https://mlsanity.com/paper/1710.03059/) (2017)
- [Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture](https://mlsanity.com/paper/2103.03330/) (2021)
- [PyTorch-Direct: Enabling GPU Centric Data Access for Very Large Graph Neural Network Training with Irregular Accesses](https://mlsanity.com/paper/2101.07956/) (2021)

## Built on by (most cited first, 16 of 16)

- [Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts](https://mlsanity.com/paper/2502.19811/) (2025)
- [Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective](https://mlsanity.com/paper/2311.13279/) (2023)
- [Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping](https://mlsanity.com/paper/2409.15241/) (2024)
- [GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism](https://mlsanity.com/paper/2303.13775/) (2023)
- [FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Training](https://mlsanity.com/paper/2301.07482/) (2023)
- [GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism](https://mlsanity.com/paper/2308.10087/) (2023)
- [Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputation](https://mlsanity.com/paper/2501.08547/) (2025)
- [Single-GPU GNN Systems: Traps and Pitfalls](https://mlsanity.com/paper/2402.03548/) (2024)
- [Deal: Distributed End-to-End GNN Inference for All Nodes](https://mlsanity.com/paper/2503.02960/) (2025)
- [Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training](https://mlsanity.com/paper/2505.04083/) (2025)
- [Efficient AllReduce with Stragglers](https://mlsanity.com/paper/2505.23523/) (2025)
- [CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs](https://mlsanity.com/paper/2605.08467/) (2026)
- [Slicing Input Features to Accelerate Deep Learning: A Case Study with Graph Neural Networks](https://mlsanity.com/paper/2408.11500/) (2024)
- [Grappa: Gradient-Only Communication for Scalable Graph Neural Network Training](https://mlsanity.com/paper/2602.01872/) (2026)
- [FeLoG: Scalable and Efficient Distributed Graph Embedding with Feedback Loop Mechanism](https://mlsanity.com/paper/2606.22180/) (2026)
- [ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform](https://mlsanity.com/paper/2607.26566/) (2026)

## Abstract (arXiv)

The increasing size of input graphs for graph neural networks (GNNs) highlights the demand for using multi-GPU platforms. However, existing multi-GPU GNN systems optimize the computation and communication individually based on the conventional practice of scaling dense DNNs. For irregularly sparse and fine-grained GNN workloads, such solutions miss the opportunity to jointly schedule/optimize the computation and communication operations for high-performance delivery. To this end, we propose MGG, a novel system design to accelerate full-graph GNNs on multi-GPU platforms. The core of MGG is its novel dynamic software pipeline to facilitate fine-grained computation-communication overlapping within a GPU kernel. Specifically, MGG introduces GNN-tailored pipeline construction and GPU-aware pipeline mapping to facilitate workload balancing and operation overlapping. MGG also incorporates an intelligent runtime design with analytical modeling and optimization heuristics to dynamically improve the execution performance. Extensive evaluation reveals that MGG outperforms state-of-the-art full-graph GNN systems across various settings: on average 4.41X, 4.81X, and 10.83X faster than DGL, MGG-UVM, and ROC, respectively.