# The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey

arXiv: https://arxiv.org/abs/2305.13854 · HTML: https://mlsanity.com/paper/2305.13854/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 99 of its 225 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 99)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Residual Gated Graph ConvNets](https://mlsanity.com/paper/1711.07553/) (2017)

## Built on by (most cited first, 14 of 14)

- [Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective](https://mlsanity.com/paper/2311.13279/) (2023)
- [Forget Many, Forget Right: Scalable and Precise Concept Unlearning in Diffusion Models](https://mlsanity.com/paper/2601.06162/) (2026)
- [An Experimental Comparison of Partitioning Strategies for Distributed Graph Neural Network Training](https://mlsanity.com/paper/2308.15602/) (2023)
- [Graph Neural Network-Based Collaborative Perception for Adaptive Scheduling in Distributed Systems](https://mlsanity.com/paper/2505.16248/) (2025)
- [Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking](https://mlsanity.com/paper/2601.06163/) (2026)
- [Limited Communications Distributed Optimization via Deep Unfolded Distributed ADMM](https://mlsanity.com/paper/2309.14353/) (2023)
- [D3-GNN: Dynamic Distributed Dataflow for Streaming Graph Neural Networks](https://mlsanity.com/paper/2409.09079/) (2024)
- [Graph Retention Networks for Dynamic Graphs](https://mlsanity.com/paper/2411.11259/) (2024)
- [AMUN: Adversarial Machine UNlearning](https://mlsanity.com/paper/2503.00917/) (2025)
- [WARP: Weight Teleportation for Attack-Resilient Unlearning Protocols](https://mlsanity.com/paper/2512.00272/) (2025)
- [Factored Value Functions for Graph-Based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.11401/) (2026)
- [Vision Paper: Designing Graph Neural Networks in Compliance with the European Artificial Intelligence Act](https://mlsanity.com/paper/2410.22120/) (2024)
- [Fair Minimum Labeling: Efficient Temporal Network Activations for Reachability and Equity](https://mlsanity.com/paper/2510.03899/) (2025)
- [FedSGT: Exact Federated Unlearning via Sequential Group-based Training](https://mlsanity.com/paper/2511.23393/) (2025)

## Abstract (arXiv)

Graph Neural Networks (GNNs) are an emerging research field. This specialized Deep Neural Network (DNN) architecture is capable of processing graph structured data and bridges the gap between graph processing and Deep Learning (DL). As graphs are everywhere, GNNs can be applied to various domains including recommendation systems, computer vision, natural language processing, biology and chemistry. With the rapid growing size of real world graphs, the need for efficient and scalable GNN training solutions has come. Consequently, many works proposing GNN systems have emerged throughout the past few years. However, there is an acute lack of overview, categorization and comparison of such systems. We aim to fill this gap by summarizing and categorizing important methods and techniques for large-scale GNN solutions. In addition, we establish connections between GNN systems, graph processing systems and DL systems.