# Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning

arXiv: https://arxiv.org/abs/2410.17494 · HTML: https://mlsanity.com/paper/2410.17494/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 91 references that are in the map.
- First posted on arXiv 23 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks](https://mlsanity.com/paper/2107.13048/) (2021)
- [AM-GCN: Adaptive Multi-channel Graph Convolutional Networks](https://mlsanity.com/paper/2007.02265/) (2020)
- [Cross-Modal Translation and Alignment for Survival Analysis](https://mlsanity.com/paper/2309.12855/) (2023)
- [SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design](https://mlsanity.com/paper/2401.16456/) (2024)
- [Multimodal Fusion on Low-quality Data: A Comprehensive Survey](https://mlsanity.com/paper/2404.18947/) (2024)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [Multi-modal Graph Learning for Disease Prediction](https://mlsanity.com/paper/2203.05880/) (2022)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [ImageGCN: Multi-Relational Image Graph Convolutional Networks for Disease Identification with Chest X-rays](https://mlsanity.com/paper/1904.00325/) (2019)
- [Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach](https://mlsanity.com/paper/2403.19067/) (2024)
- [ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressions](https://mlsanity.com/paper/2303.12364/) (2023)
- [Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Prediction](https://mlsanity.com/paper/2308.15469/) (2023)
- [Gradformer: Graph Transformer with Exponential Decay](https://mlsanity.com/paper/2404.15729/) (2024)
- [GTC: GNN-Transformer Co-contrastive Learning for Self-supervised Heterogeneous Graph Representation](https://mlsanity.com/paper/2403.15520/) (2024)
- [Cluster-wise Graph Transformer with Dual-granularity Kernelized Attention](https://mlsanity.com/paper/2410.06746/) (2024)
- [AMA-GCN: Adaptive Multi-layer Aggregation Graph Convolutional Network for Disease Prediction](https://mlsanity.com/paper/2106.08732/) (2021)
- [Graph External Attention Enhanced Transformer](https://mlsanity.com/paper/2405.21061/) (2024)
- [Co-Attentive Cross-Modal Deep Learning for Medical Evidence Synthesis and Decision Making](https://mlsanity.com/paper/1909.06442/) (2019)
- [Dopamine Transporter SPECT Image Classification for Neurodegenerative Parkinsonism via Diffusion Maps and Machine Learning Classifiers](https://mlsanity.com/paper/2104.02066/) (2021)
- [TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction](https://mlsanity.com/paper/2307.07177/) (2023)
- [Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification](https://mlsanity.com/paper/2312.04189/) (2023)

## Abstract (arXiv)

The classification of medical images is a pivotal aspect of disease diagnosis, often enhanced by deep learning techniques. However, traditional approaches typically focus on unimodal medical image data, neglecting the integration of diverse non-image patient data. This paper proposes a novel Cross-Graph Modal Contrastive Learning (CGMCL) framework for multimodal structured data from different data domains to improve medical image classification. The model effectively integrates both image and non-image data by constructing cross-modality graphs and leveraging contrastive learning to align multimodal features in a shared latent space. An inter-modality feature scaling module further optimizes the representation learning process by reducing the gap between heterogeneous modalities. The proposed approach is evaluated on two datasets: a Parkinson's disease (PD) dataset and a public melanoma dataset. Results demonstrate that CGMCL outperforms conventional unimodal methods in accuracy, interpretability, and early disease prediction. Additionally, the method shows superior performance in multi-class melanoma classification. The CGMCL framework provides valuable insights into medical image classification while offering improved disease interpretability and predictive capabilities.