# MM-GTUNets: Unified Multi-Modal Graph Deep Learning for Brain Disorders Prediction

arXiv: https://arxiv.org/abs/2406.14455 · HTML: https://mlsanity.com/paper/2406.14455/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Disease](https://mlsanity.com/paper/1806.01738/) (2018)
- [Graph Neural Networks in Network Neuroscience](https://mlsanity.com/paper/2106.03535/) (2021)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [Differentiable Graph Module (DGM) for Graph Convolutional Networks](https://mlsanity.com/paper/2002.04999/) (2020)
- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [Multimodal Fusion on Low-quality Data: A Comprehensive Survey](https://mlsanity.com/paper/2404.18947/) (2024)
- [InceptionGCN: Receptive Field Aware Graph Convolutional Network for Disease Prediction](https://mlsanity.com/paper/1903.04233/) (2019)
- [Latent-Graph Learning for Disease Prediction](https://mlsanity.com/paper/2003.13620/) (2020)
- [Multi-modal Graph Learning for Disease Prediction](https://mlsanity.com/paper/2203.05880/) (2022)
- [GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis](https://mlsanity.com/paper/2203.09034/) (2022)
- [Predictive Dynamic Fusion](https://mlsanity.com/paper/2406.04802/) (2024)
- [Unsupervised pre-training of graph transformers on patient population graphs](https://mlsanity.com/paper/2207.10603/) (2022)
- [Graph-in-Graph (GiG): Learning interpretable latent graphs in non-Euclidean domain for biological and healthcare applications](https://mlsanity.com/paper/2204.00323/) (2022)
- [How good are variational autoencoders at transfer learning?](https://mlsanity.com/paper/2304.10767/) (2023)
- [Variational Transfer Learning using Cross-Domain Latent Modulation](https://mlsanity.com/paper/2205.15523/) (2022)

## Built on by (most cited first, 9 of 9)

- [MHNet: Multi-view High-order Network for Diagnosing Neurodevelopmental Disorders Using Resting-state fMRI](https://mlsanity.com/paper/2407.03217/) (2024)
- [Information Bottleneck-Guided Heterogeneous Graph Learning for Interpretable Neurodevelopmental Disorder Diagnosis](https://mlsanity.com/paper/2502.20769/) (2025)
- [PIME: Prototype-based Interpretable MCTS-Enhanced Brain Network Analysis for Disorder Diagnosis](https://mlsanity.com/paper/2602.21046/) (2026)
- [LEL: Lipschitz Continuity Constrained Ensemble Learning for Efficient EEG-Based Intra-subject Emotion Recognition](https://mlsanity.com/paper/2504.09156/) (2025)
- [Brain-HGCN: A Hyperbolic Graph Convolutional Network for Brain Functional Network Analysis](https://mlsanity.com/paper/2509.14965/) (2025)
- [Fine-Grained Traceability for Transparent ML Pipelines](https://mlsanity.com/paper/2601.14971/) (2026)
- [A Multimodal Deep Learning Approach for White Matter Shape Prediction in Diffusion MRI Tractography](https://mlsanity.com/paper/2504.18400/) (2025)
- [fMRI-Diffusion: Generating fMRI Time Series Via a Temporal Transformer Diffusion Model for Major Depressive Disorder Diagnosis](https://mlsanity.com/paper/2605.24065/) (2026)
- [M2LG-DG: A Multi-modal Local-Global Domain Generalization Framework for Cross-site Major Depressive Disorder Classification](https://mlsanity.com/paper/2609.09186/) (2026)

## Abstract (arXiv)

Graph deep learning (GDL) has demonstrated impressive performance in predicting population-based brain disorders (BDs) through the integration of both imaging and non-imaging data. However, the effectiveness of GDL based methods heavily depends on the quality of modeling the multi-modal population graphs and tends to degrade as the graph scale increases. Furthermore, these methods often constrain interactions between imaging and non-imaging data to node-edge interactions within the graph, overlooking complex inter-modal correlations, leading to suboptimal outcomes. To overcome these challenges, we propose MM-GTUNets, an end-to-end graph transformer based multi-modal graph deep learning (MMGDL) framework designed for brain disorders prediction at large scale. Specifically, to effectively leverage rich multi-modal information related to diseases, we introduce Modality Reward Representation Learning (MRRL) which adaptively constructs population graphs using a reward system. Additionally, we employ variational autoencoder to reconstruct latent representations of non-imaging features aligned with imaging features. Based on this, we propose Adaptive Cross-Modal Graph Learning (ACMGL), which captures critical modality-specific and modality-shared features through a unified GTUNet encoder taking advantages of Graph UNet and Graph Transformer, and feature fusion module. We validated our method on two public multi-modal datasets ABIDE and ADHD-200, demonstrating its superior performance in diagnosing BDs. Our code is available at https://github.com/NZWANG/MM-GTUNets.