mlsanityData to 13 Oct 2026

Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning

arXiv 2410.17494 ↗First posted 23 Oct 2024eess.IV cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The classification of medical images is a pivotal aspect of disease diagnosis, often enhanced by deep learning techniques. However, traditional approaches typically focus on unimodal medical image data, neglecting the integration of diverse non-image patient data. This paper proposes a novel Cross-Graph Modal Contrastive Learning (CGMCL) framework for multimodal structured data from different data domains to improve medical image classification. The model effectively integrates both image and non-image data by constructing cross-modality graphs and leveraging contrastive learning to align multimodal features in a shared latent space. An inter-modality feature scaling module further optimizes the representation learning process by reducing the gap between heterogeneous modalities. The proposed approach is evaluated on two datasets: a Parkinson's disease (PD) dataset and a public melanoma dataset. Results demonstrate that CGMCL outperforms conventional unimodal methods in accuracy, interpretability, and early disease prediction. Additionally, the method shows superior performance in multi-class melanoma classification. The CGMCL framework provides valuable insights into medical image classification while offering improved disease interpretability and predictive capabilities.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 29 of its 91 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

29 of 91 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,318
  4. 2017Graph Attention Networkscited by 8,888
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  6. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 514
  7. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  8. 2021Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networkscited by 124
  9. 2020AM-GCN: Adaptive Multi-channel Graph Convolutional Networkscited by 121
  10. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  11. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 55
  12. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
17 more
  1. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
  2. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 35
  3. 2022Multi-modal Graph Learning for Disease Predictioncited by 34
  4. 2024A review of deep learning-based information fusion techniques for multimodal medical image classificationcited by 25
  5. 2019ImageGCN: Multi-Relational Image Graph Convolutional Networks for Disease Identification with Chest X-rayscited by 22
  6. 2024Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approachcited by 18
  7. 2023ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressionscited by 12
  8. 2023Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Predictioncited by 12
  9. 2024Gradformer: Graph Transformer with Exponential Decaycited by 10
  10. 2024GTC: GNN-Transformer Co-contrastive Learning for Self-supervised Heterogeneous Graph Representationcited by 9
  11. 2024Cluster-wise Graph Transformer with Dual-granularity Kernelized Attentioncited by 8
  12. 2021AMA-GCN: Adaptive Multi-layer Aggregation Graph Convolutional Network for Disease Predictioncited by 4
  13. 2024Graph External Attention Enhanced Transformercited by 3
  14. 2019Co-Attentive Cross-Modal Deep Learning for Medical Evidence Synthesis and Decision Makingcited by 2
  15. 2021Dopamine Transporter SPECT Image Classification for Neurodegenerative Parkinsonism via Diffusion Maps and Machine Learning Classifierscited by 2
  16. 2023TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Predictioncited by 2
  17. 2023Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classificationcited by 2

Built on by

Not cited yet.