mlsanityData to 7 Oct 2026

GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation

arXiv 2407.05540 ↗First posted 8 Jul 2024cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent advances in learning multi-modal representation have witnessed the success in biomedical domains. While established techniques enable handling multi-modal information, the challenges are posed when extended to various clinical modalities and practical modalitymissing setting due to the inherent modality gaps. To tackle these, we propose an innovative Modality-prompted Heterogeneous Graph for Omnimodal Learning (GTP-4o), which embeds the numerous disparate clinical modalities into a unified representation, completes the deficient embedding of missing modality and reformulates the cross-modal learning with a graph-based aggregation. Specially, we establish a heterogeneous graph embedding to explicitly capture the diverse semantic properties on both the modality-specific features (nodes) and the cross-modal relations (edges). Then, we design a modality-prompted completion that enables completing the inadequate graph representation of missing modality through a graph prompting mechanism, which generates hallucination graphic topologies to steer the missing embedding towards the intact representation. Through the completed graph, we meticulously develop a knowledge-guided hierarchical cross-modal aggregation consisting of a global meta-path neighbouring to uncover the potential heterogeneous neighbors along the pathways driven by domain knowledge, and a local multi-relation aggregation module for the comprehensive cross-modal interaction across various heterogeneous relations. We assess the efficacy of our methodology on rigorous benchmarking experiments against prior state-of-the-arts. In a nutshell, GTP-4o presents an initial foray into the intriguing realm of embedding, relating and perceiving the heterogeneous patterns from various clinical modalities holistically via a graph theory. Project page: https://gtp-4-o.github.io/.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 47 of its 87 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

47 of 87 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  6. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2020Heterogeneous Graph Transformercited by 581
  10. 2019Graph Transformer Networkscited by 520
  11. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  12. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
35 more
  1. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  2. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  3. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  4. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  5. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  6. 2021Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networkscited by 124
  7. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  8. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 113
  9. 2023Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Predictioncited by 86
  10. 2020A review: Deep learning for medical image segmentation using multi-modality fusioncited by 84
  11. 2022A graph-transformer for whole slide image classificationcited by 84
  12. 2022SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detectioncited by 74
  13. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  14. 2022Dynamic Multimodal Fusioncited by 54
  15. 2024Endora: Video Generation Models as Endoscopy Simulatorscited by 53
  16. 2022Transformer-based Multimodal Information Fusion for Facial Expression Analysiscited by 49
  17. 2020Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embeddingcited by 48
  18. 2020Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosiscited by 47
  19. 2021Modality-aware Mutual Learning for Multi-modal Medical Image Segmentationcited by 47
  20. 2023Histopathology Whole Slide Image Analysis with Heterogeneous Graph Representation Learningcited by 47
  21. 2021Domain Generalization on Medical Imaging Classification using Episodic Training with Task Augmentationcited by 33
  22. 2022StegaNeRF: Embedding Invisible Information within Neural Radiance Fieldscited by 32
  23. 2024LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstructioncited by 21
  24. 2022Towards Robust Adaptive Object Detection under Noisy Annotationscited by 20
  25. 2022Knowledge Condensation Distillationcited by 20
  26. 2022Heterogeneous Graph Learning for Multi-modal Medical Data Analysiscited by 16
  27. 2024EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splattingcited by 16
  28. 2021Consistent Posterior Distributions under Vessel-Mixing: A Regularization for Cross-Domain Retinal Artery/Vein Classificationcited by 15
  29. 2021Hierarchical Deep Network with Uncertainty-aware Semi-supervised Learning for Vessel Segmentationcited by 15
  30. 2018Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancercited by 14
  31. 2020Generator Versus Segmentor: Pseudo-healthy Synthesiscited by 12
  32. 2022AFSC: Adaptive Fourier Space Compression for Anomaly Detectioncited by 12
  33. 2024GaussianStego: A Generalizable Stenography Pipeline for Generative 3D Gaussians Splattingcited by 11
  34. 2021Unsupervised Anomaly Segmentation using Image-Semantic Cycle Translationcited by 10
  35. 2020Online Disease Self-diagnosis with Inductive Heterogeneous Graph Convolutional Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2025Distilled Prompt Learning for Incomplete Multimodal Survival Predictioncited by 10
  2. 2024NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentationcited by 5
  3. 2024When 3D Partial Points Meets SAM: Tooth Point Cloud Segmentation with Sparse Labelscited by 5
  4. 2024HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adaptercited by 3
  5. 2026Beyond Sparse Supervision: Diffusion-Guided Learning for Few-Shot Graph Fraud Detectioncited by 2
  6. 2025DG-STMTL: A Novel Graph Convolutional Network for Multi-Task Spatio-Temporal Traffic Forecastingcited by 0
  7. 2026FU-Mamba: A Frequency-Enhanced Dynamic Scanning Framework for Oralscan Image Segmentationcited by 0