# Multi-modal learning for predicting the genotype of glioma

arXiv: https://arxiv.org/abs/2203.10852 · HTML: https://mlsanity.com/paper/2203.10852/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [GNNExplainer: Generating Explanations for Graph Neural Networks](https://mlsanity.com/paper/1903.03894/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Disease](https://mlsanity.com/paper/1806.01738/) (2018)
- [Generating Diffusion MRI scalar maps from T1 weighted images using generative adversarial networks](https://mlsanity.com/paper/1810.02683/) (2018)
- [Collaborative learning of images and geometrics for predicting isocitrate dehydrogenase status of glioma](https://mlsanity.com/paper/2201.05530/) (2022)
- [Predicting isocitrate dehydrogenase mutation status in glioma using structural brain networks and graph neural networks](https://mlsanity.com/paper/2109.01854/) (2021)

## Built on by (most cited first, 15 of 15)

- [CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2303.14081/) (2023)
- [DisC-Diff: Disentangled Conditional Diffusion Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2303.13933/) (2023)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Phy-Diff: Physics-guided Hourglass Diffusion Model for Diffusion MRI Synthesis](https://mlsanity.com/paper/2406.03002/) (2024)
- [Cross-modal Diffusion Modelling for Super-resolved Spatial Transcriptomics](https://mlsanity.com/paper/2404.12973/) (2024)
- [Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning](https://mlsanity.com/paper/2406.13979/) (2024)
- [D-CoRP: Differentiable Connectivity Refinement for Functional Brain Networks](https://mlsanity.com/paper/2405.18658/) (2024)
- [Spatiotemporal Graph Neural Network Modelling Perfusion MRI](https://mlsanity.com/paper/2406.06434/) (2024)
- [Genomics-guided Representation Learning for Pathologic Pan-cancer Tumor Microenvironment Subtype Prediction](https://mlsanity.com/paper/2406.06517/) (2024)
- [Complementary Information Mutual Learning for Multimodality Medical Image Segmentation](https://mlsanity.com/paper/2401.02717/) (2024)
- [Unified Modeling Enhanced Multimodal Learning for Precision Neuro-Oncology](https://mlsanity.com/paper/2406.07078/) (2024)
- [Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling](https://mlsanity.com/paper/2408.00311/) (2024)
- [Diagnostic performance of deep learning for predicting glioma isocitrate dehydrogenase and 1p/19q co-deletion in MRI: a systematic review and meta-analysis](https://mlsanity.com/paper/2411.02426/) (2024)
- [Joint Modelling Histology and Molecular Markers for Cancer Classification](https://mlsanity.com/paper/2502.07979/) (2025)

## Abstract (arXiv)

The isocitrate dehydrogenase (IDH) gene mutation is an essential biomarker for the diagnosis and prognosis of glioma. It is promising to better predict glioma genotype by integrating focal tumor image and geometric features with brain network features derived from MRI. Convolutions neural networks show reasonable performance in predicting IDH mutation, which, however, cannot learn from non-Euclidean data, e.g., geometric and network data. In this study, we propose a multi-modal learning framework using three separate encoders to extract features of focal tumor image, tumor geometrics and global brain networks. To mitigate the limited availability of diffusion MRI, we develop a self-supervised approach to generate brain networks from anatomical multi-sequence MRI. Moreover, to extract tumor-related features from the brain network, we design a hierarchical attention module for the brain network encoder. Further, we design a bi-level multi-modal contrastive loss to align the multi-modal features and tackle the domain gap at the focal tumor and global brain. Finally, we propose a weighted population graph to integrate the multi-modal features for genotype prediction. Experimental results on the testing set show that the proposed model outperforms the baseline deep learning models. The ablation experiments validate the performance of different components of the framework. The visualized interpretation corresponds to clinical knowledge with further validation. In conclusion, the proposed learning framework provides a novel approach for predicting the genotype of glioma.