mlsanityData to 7 Oct 2026

MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators

arXiv 2501.01951 ↗First posted 3 Jan 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Graph convolutional networks (GCNs) have demonstrated superiority in graph-based learning tasks. However, training GCNs on full graphs is particularly challenging, due to the following two challenges: (1) the associated feature tensors can easily explode the memory and block the communication bandwidth of modern accelerators, and (2) the computation workflow in training GCNs alternates between sparse and dense matrix operations, complicating the efficient utilization of computational resources. Existing solutions for scalable distributed full-graph GCN training mostly adopt partition parallelism, which is unsatisfactory as they only partially address the first challenge while incurring scaled-out communication volume. To this end, we propose MixGCN aiming to simultaneously address both the aforementioned challenges towards GCN training. To tackle the first challenge, MixGCN integrates mixture of parallelism. Both theoretical and empirical analysis verify its constant communication volumes and enhanced balanced workload; For handling the second challenge, we consider mixture of accelerators (i.e., sparse and dense accelerators) with a dedicated accelerator for GCN training and a fine-grain pipeline. Extensive experiments show that MixGCN achieves boosted training efficiency and scalability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 91 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

50 of 91 references.

References · 50

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2017Graph Attention Networkscited by 8,854
  3. 2017Inductive Representation Learning on Large Graphscited by 6,284
  4. 2018How Powerful are Graph Neural Networks?cited by 4,355
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  12. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
38 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  4. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  5. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  6. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 152
  7. 2021Graph Learning: A Surveycited by 140
  8. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  9. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  10. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  11. 2021GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddingscited by 90
  12. 2017Device Placement Optimization with Reinforcement Learningcited by 84
  13. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  14. 2019Graph Representation Learning: A Surveycited by 75
  15. 2020HyGCN: A GCN Accelerator with Hybrid Architecturecited by 73
  16. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  17. 2020GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUscited by 51
  18. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  19. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  20. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  21. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  22. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  23. 2019EnGN: A High-Throughput and Energy-Efficient Accelerator for Large Graph Neural Networkscited by 42
  24. 2020GRIP: A Graph Neural Network Accelerator Architecturecited by 37
  25. 2019GraphACT: Accelerating GCN Training on CPU-FPGA Heterogeneous Platformscited by 29
  26. 2021Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspectivecited by 28
  27. 2020Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insightscited by 27
  28. 2020FusedMM: A Unified SDDMM-SpMM Kernel for Graph Embedding and Graph Neural Networkscited by 26
  29. 2022Data-Efficient Graph Grammar Learning for Molecular Generationcited by 26
  30. 2021Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networkscited by 24
  31. 2024Towards Cognitive AI Systems: a Survey and Prospective on Neuro-Symbolic AIcited by 24
  32. 2022I-GCN: A Graph Convolutional Network Accelerator with Runtime Locality Enhancement through Islandizationcited by 21
  33. 2021G-CoS: GNN-Accelerator Co-Search Towards Both Better Accuracy and Efficiencycited by 15
  34. 2021GCoD: Graph Convolutional Network Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 12
  35. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  36. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  37. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  38. 2022Distributed Graph Neural Network Training with Periodic Stale Representation Synchronizationcited by 3

Built on by

Not cited yet.