# gSuite: A Flexible and Framework Independent Benchmark Suite for Graph Neural Network Inference on GPUs

arXiv: https://arxiv.org/abs/2210.11601 · HTML: https://mlsanity.com/paper/2210.11601/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 79 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks](https://mlsanity.com/paper/2007.02901/) (2020)
- [A Fair Comparison of Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/1912.09893/) (2019)
- [Adaptive Graph Convolutional Neural Networks](https://mlsanity.com/paper/1801.03226/) (2018)
- [HyGCN: A GCN Accelerator with Hybrid Architecture](https://mlsanity.com/paper/2001.02514/) (2020)
- [Theano: A Python framework for fast computation of mathematical expressions](https://mlsanity.com/paper/1605.02688/) (2016)
- [Graph Neural Networks in TensorFlow and Keras with Spektral](https://mlsanity.com/paper/2006.12138/) (2020)
- [GRIP: A Graph Neural Network Accelerator Architecture](https://mlsanity.com/paper/2007.13828/) (2020)
- [One-pixel Signature: Characterizing CNN Models for Backdoor Detection](https://mlsanity.com/paper/2008.07711/) (2020)
- [Pooling Regularized Graph Neural Network for fMRI Biomarker Analysis](https://mlsanity.com/paper/2007.14589/) (2020)
- [Characterizing and Understanding GCNs on GPU](https://mlsanity.com/paper/2001.10160/) (2020)
- [Graph Partition Neural Networks for Semi-Supervised Classification](https://mlsanity.com/paper/1803.06272/) (2018)
- [Gunrock: A High-Performance Graph Processing Library on the GPU](https://mlsanity.com/paper/1501.05387/) (2015)
- [Loyalty in Online Communities](https://mlsanity.com/paper/1703.03386/) (2017)
- [Architectural Implications of Graph Neural Networks](https://mlsanity.com/paper/2009.00804/) (2020)
- [Analyzing Machine Learning Workloads Using a Detailed GPU Simulator](https://mlsanity.com/paper/1811.08933/) (2018)
- [GraphLab: A New Framework For Parallel Machine Learning](https://mlsanity.com/paper/1408.2041/) (2014)
- [Message-Passing Neural Networks Learn Little's Law](https://mlsanity.com/paper/1901.05748/) (2019)

## Abstract (arXiv)

As the interest to Graph Neural Networks (GNNs) is growing, the importance of benchmarking and performance characterization studies of GNNs is increasing. So far, we have seen many studies that investigate and present the performance and computational efficiency of GNNs. However, the work done so far has been carried out using a few high-level GNN frameworks. Although these frameworks provide ease of use, they contain too many dependencies to other existing libraries. The layers of implementation details and the dependencies complicate the performance analysis of GNN models that are built on top of these frameworks, especially while using architectural simulators. Furthermore, different approaches on GNN computation are generally overlooked in prior characterization studies, and merely one of the common computational models is evaluated. Based on these shortcomings and needs that we observed, we developed a benchmark suite that is framework independent, supporting versatile computational models, easily configurable and can be used with architectural simulators without additional effort. Our benchmark suite, which we call gSuite, makes use of only hardware vendor's libraries and therefore it is independent of any other frameworks. gSuite enables performing detailed performance characterization studies on GNN Inference using both contemporary GPU profilers and architectural GPU simulators. To illustrate the benefits of our new benchmark suite, we perform a detailed characterization study with a set of well-known GNN models with various datasets; running gSuite both on a real GPU card and a timing-detailed GPU simulator. We also implicate the effect of computational models on performance. We use several evaluation metrics to rigorously measure the performance of GNN computation.