# Heterogeneous Graph-Based Multimodal Brain Network Learning

arXiv: https://arxiv.org/abs/2110.08465 · HTML: https://mlsanity.com/paper/2110.08465/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 99 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning](https://mlsanity.com/paper/2105.09111/) (2021)
- [A Survey of Heterogeneous Information Network Analysis](https://mlsanity.com/paper/1511.04854/) (2015)
- [Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Disease](https://mlsanity.com/paper/1806.01738/) (2018)
- [Spatio-Temporal Graph Convolution for Resting-State fMRI Analysis](https://mlsanity.com/paper/2003.10613/) (2020)
- [Understanding Graph Isomorphism Network for rs-fMRI Functional Connectivity Analysis](https://mlsanity.com/paper/2001.03690/) (2020)
- [Graph Neural Network for Interpreting Task-fMRI Biomarkers](https://mlsanity.com/paper/1907.01661/) (2019)
- [InceptionGCN: Receptive Field Aware Graph Convolutional Network for Disease Prediction](https://mlsanity.com/paper/1903.04233/) (2019)
- [A Graph Neural Network Framework for Causal Inference in Brain Networks](https://mlsanity.com/paper/2010.07143/) (2020)
- [Deep Representation Learning For Multimodal Brain Networks](https://mlsanity.com/paper/2007.09777/) (2020)
- [Emotion Recognition with Machine Learning Using EEG Signals](https://mlsanity.com/paper/1903.07272/) (2019)
- [Multi-view Graph Embedding with Hub Detection for Brain Network Analysis](https://mlsanity.com/paper/1709.03659/) (2017)
- [MGN-Net: a multi-view graph normalizer for integrating heterogeneous biological network populations](https://mlsanity.com/paper/2104.03895/) (2021)

## Built on by (most cited first, 5 of 5)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [Generative AI Enables Structural Brain Network Construction from fMRI via Symmetric Diffusion Learning](https://mlsanity.com/paper/2309.16205/) (2023)
- [COMMA: Coordinate-aware Modulated Mamba Network for 3D Dispersed Vessel Segmentation](https://mlsanity.com/paper/2503.02332/) (2025)
- [A Brain-to-Population Graph Learning Framework for Diagnosing Brain Disorders](https://mlsanity.com/paper/2506.16096/) (2025)
- [Exploring Subnetwork Interactions in Heterogeneous Brain Network via Prior-Informed Graph Learning](https://mlsanity.com/paper/2603.19307/) (2026)

## Abstract (arXiv)

Graph neural networks (GNNs) provide powerful insights for brain neuroimaging technology from the view of graphical networks. However, most existing GNN-based models assume that the neuroimaging-produced brain connectome network is a homogeneous graph with single types of nodes and edges. In fact, emerging studies have reported and emphasized the significance of heterogeneity among human brain activities, especially between the two cerebral hemispheres. Thus, homogeneous-structured brain network-based graph methods are insufficient for modelling complicated cerebral activity states. To overcome this problem, in this paper, we present a heterogeneous graph neural network (HebrainGNN) for multimodal brain neuroimaging fusion learning. We first model the brain network as a heterogeneous graph with multitype nodes (i.e., left and right hemispheric nodes) and multitype edges (i.e., intra- and interhemispheric edges). Then, we propose a self-supervised pretraining strategy based on a heterogeneous brain network to address the potential overfitting problem caused by the conflict between a large parameter size and a small medical data sample size. Our results show the superiority of the proposed model over other existing methods in brain-related disease prediction tasks. Ablation experiments show that our heterogeneous graph-based model attaches more importance to hemispheric connections that may be neglected due to their low strength by previous homogeneous graph models. Other experiments also indicate that our proposed model with a pretraining strategy alleviates the problem of limited labelled data and yields a significant improvement in accuracy.