# Deep Ensembles for Graphs with Higher-order Dependencies

arXiv: https://arxiv.org/abs/2205.13988 · HTML: https://mlsanity.com/paper/2205.13988/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 30 of 33)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks](https://mlsanity.com/paper/1810.02244/) (2018)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Higher-Order Explanations of Graph Neural Networks via Relevant Walks](https://mlsanity.com/paper/2006.03589/) (2020)
- [Networks beyond pairwise interactions: structure and dynamics](https://mlsanity.com/paper/2006.01764/) (2020)
- [Higher-order organization of complex networks](https://mlsanity.com/paper/1612.08447/) (2016)
- [Decoupling the Depth and Scope of Graph Neural Networks](https://mlsanity.com/paper/2201.07858/) (2022)
- [The physics of higher-order interactions in complex systems](https://mlsanity.com/paper/2110.06023/) (2021)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [Evaluating Link Prediction Methods](https://mlsanity.com/paper/1505.04094/) (2015)
- [The why, how, and when of representations for complex systems](https://mlsanity.com/paper/2006.02870/) (2020)
- [Memory in network flows and its effects on spreading dynamics and community detection](https://mlsanity.com/paper/1305.4807/) (2013)
- [Graph Ensemble Learning over Multiple Dependency Trees for Aspect-level Sentiment Classification](https://mlsanity.com/paper/2103.11794/) (2021)
- [Calendar Graph Neural Networks for Modeling Time Structures in Spatiotemporal User Behaviors](https://mlsanity.com/paper/2006.06820/) (2020)
- [Causality-Driven Slow-Down and Speed-Up of Diffusion in Non-Markovian Temporal Networks](https://mlsanity.com/paper/1307.4030/) (2013)
- [Representing higher-order dependencies in networks](https://mlsanity.com/paper/1508.03113/) (2015)
- [Modeling sequences and temporal networks with dynamic community structures](https://mlsanity.com/paper/1509.04740/) (2015)

## Built on by (most cited first, 3 of 3)

- [De Bruijn goes Neural: Causality-Aware Graph Neural Networks for Time Series Data on Dynamic Graphs](https://mlsanity.com/paper/2209.08311/) (2022)
- [Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?](https://mlsanity.com/paper/2605.22593/) (2026)
- [Representing Outcome-driven Higher-order Dependencies in Graphs of Disease Trajectories](https://mlsanity.com/paper/2312.15353/) (2023)

## Abstract (arXiv)

Graph neural networks (GNNs) continue to achieve state-of-the-art performance on many graph learning tasks, but rely on the assumption that a given graph is a sufficient approximation of the true neighborhood structure. When a system contains higher-order sequential dependencies, we show that the tendency of traditional graph representations to underfit each node's neighborhood causes existing GNNs to generalize poorly. To address this, we propose a novel Deep Graph Ensemble (DGE), which captures neighborhood variance by training an ensemble of GNNs on different neighborhood subspaces of the same node within a higher-order network structure. We show that DGE consistently outperforms existing GNNs on semisupervised and supervised tasks on six real-world data sets with known higher-order dependencies, even under a similar parameter budget. We demonstrate that learning diverse and accurate base classifiers is central to DGE's success, and discuss the implications of these findings for future work on ensembles of GNNs.