# Beyond Real-world Benchmark Datasets: An Empirical Study of Node Classification with GNNs

arXiv: https://arxiv.org/abs/2206.09144 · HTML: https://mlsanity.com/paper/2206.09144/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 18 Jun 2022.

## Builds on (most cited first, 30 of 39)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Geom-GCN: Geometric Graph Convolutional Networks](https://mlsanity.com/paper/2002.05287/) (2020)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [Adaptive Universal Generalized PageRank Graph Neural Network](https://mlsanity.com/paper/2006.07988/) (2020)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [Community Detection and Stochastic Block Models](https://mlsanity.com/paper/1703.10146/) (2017)
- [Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods](https://mlsanity.com/paper/2110.14446/) (2021)
- [SIGN: Scalable Inception Graph Neural Networks](https://mlsanity.com/paper/2004.11198/) (2020)
- [Graph Neural Networks with Heterophily](https://mlsanity.com/paper/2009.13566/) (2020)
- [Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks](https://mlsanity.com/paper/2112.14936/) (2021)
- [Graph Neural Networks for Graphs with Heterophily: A Survey](https://mlsanity.com/paper/2202.07082/) (2022)
- [GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks](https://mlsanity.com/paper/2103.08826/) (2021)

## Built on by (most cited first, 23 of 23)

- [Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification](https://mlsanity.com/paper/2406.08993/) (2024)
- [GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data](https://mlsanity.com/paper/2409.14500/) (2024)
- [Mixture of Link Predictors on Graphs](https://mlsanity.com/paper/2402.08583/) (2024)
- [A Comprehensive Benchmark on Spectral GNNs: The Impact on Efficiency, Memory, and Effectiveness](https://mlsanity.com/paper/2406.09675/) (2024)
- [Investigating the Interplay between Features and Structures in Graph Learning](https://mlsanity.com/paper/2308.09570/) (2023)
- [Bkd-FedGNN: A Benchmark for Classification Backdoor Attacks on Federated Graph Neural Network](https://mlsanity.com/paper/2306.10351/) (2023)
- [Graph Neural Networks on Quantum Computers](https://mlsanity.com/paper/2405.17060/) (2024)
- [Revealing the Pitfalls and Re-Evaluating the Advancement of Heterophilic Graph Learning](https://mlsanity.com/paper/2409.05755/) (2024)
- [Delving into Instance-Dependent Label Noise in Graph Data: A Comprehensive Study and Benchmark](https://mlsanity.com/paper/2506.12468/) (2025)
- [Iterative Graph Neural Network Enhancement via Frequent Subgraph Mining of Explanations](https://mlsanity.com/paper/2403.07849/) (2024)
- [A Simple and Scalable Graph Neural Network for Large Directed Graphs](https://mlsanity.com/paper/2306.08274/) (2023)
- [Unifews: You Need Fewer Operations for Efficient Graph Neural Networks](https://mlsanity.com/paper/2403.13268/) (2024)
- [Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs](https://mlsanity.com/paper/2510.18473/) (2025)
- [Structural Bias Beyond Homophily: A Study of Fairness in Link Prediction](https://mlsanity.com/paper/2602.11802/) (2026)
- [Towards Unsupervised Graph Completion Learning on Graphs with Features and Structure Missing](https://mlsanity.com/paper/2309.02762/) (2023)
- [MDS-GNN: A Mutual Dual-Stream Graph Neural Network on Graphs with Incomplete Features and Structure](https://mlsanity.com/paper/2408.04845/) (2024)
- [On Measuring Unnoticeability of Graph Adversarial Attacks: Observations, New Measure, and Applications](https://mlsanity.com/paper/2501.05015/) (2025)
- [Explainable Graph Neural Architecture Search via Monte-Carlo Tree Search (Full version)](https://mlsanity.com/paper/2308.15734/) (2023)
- [Lying Graph Convolution: Learning to Lie for Node Classification Tasks](https://mlsanity.com/paper/2405.01247/) (2024)
- [Hyperbolic Benchmarking Unveils Network Topology-Feature Relationship in GNN Performance](https://mlsanity.com/paper/2406.02772/) (2024)
- [Graph as a feature: improving node classification with non-neural graph-aware logistic regression](https://mlsanity.com/paper/2411.12330/) (2024)
- [Predict, Cluster, Refine: A Joint Embedding Predictive Self-Supervised Framework for Graph Representation Learning](https://mlsanity.com/paper/2502.01684/) (2025)
- [Hyperbolic Graph Embedders for Link Prediction and Topology Reconstruction](https://mlsanity.com/paper/2608.07029/) (2026)

## Abstract (arXiv)

Graph Neural Networks (GNNs) have achieved great success on a node classification task. Despite the broad interest in developing and evaluating GNNs, they have been assessed with limited benchmark datasets. As a result, the existing evaluation of GNNs lacks fine-grained analysis from various characteristics of graphs. Motivated by this, we conduct extensive experiments with a synthetic graph generator that can generate graphs having controlled characteristics for fine-grained analysis. Our empirical studies clarify the strengths and weaknesses of GNNs from four major characteristics of real-world graphs with class labels of nodes, i.e., 1) class size distributions (balanced vs. imbalanced), 2) edge connection proportions between classes (homophilic vs. heterophilic), 3) attribute values (biased vs. random), and 4) graph sizes (small vs. large). In addition, to foster future research on GNNs, we publicly release our codebase that allows users to evaluate various GNNs with various graphs. We hope this work offers interesting insights for future research.