mlsanityData to 7 Oct 2026

A Multi-Modal Deep Learning Framework for Pan-Cancer Prognosis

arXiv 2501.07016 ↗First posted 13 Jan 2025eess.IV cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Prognostic task is of great importance as it closely related to the survival analysis of patients, the optimization of treatment plans and the allocation of resources. The existing prognostic models have shown promising results on specific datasets, but there are limitations in two aspects. On the one hand, they merely explore certain types of modal data, such as patient histopathology WSI and gene expression analysis. On the other hand, they adopt the per-cancer-per-model paradigm, which means the trained models can only predict the prognostic effect of a single type of cancer, resulting in weak generalization ability. In this paper, a deep-learning based model, named UMPSNet, is proposed. Specifically, to comprehensively understand the condition of patients, in addition to constructing encoders for histopathology images and genomic expression profiles respectively, UMPSNet further integrates four types of important meta data (demographic information, cancer type information, treatment protocols, and diagnosis results) into text templates, and then introduces a text encoder to extract textual features. In addition, the optimal transport OT-based attention mechanism is utilized to align and fuse features of different modalities. Furthermore, a guided soft mixture of experts (GMoE) mechanism is introduced to effectively address the issue of distribution differences among multiple cancer datasets. By incorporating the multi-modality of patient data and joint training, UMPSNet outperforms all SOTA approaches, and moreover, it demonstrates the effectiveness and generalization ability of the proposed learning paradigm of a single model for multiple cancer types. The code of UMPSNet is available at https://github.com/binging512/UMPSNet.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

21 of 35 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  7. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  10. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  11. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 254
  12. 2023From Sparse to Soft Mixtures of Expertscited by 245
9 more
  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  2. 2020Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networkscited by 178
  3. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  4. 2021Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networkscited by 124
  5. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  6. 2023Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Predictioncited by 86
  7. 2023DRG-LLaMA : Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patientscited by 29
  8. 2022AdvMIL: Adversarial Multiple Instance Learning for the Survival Analysis on Whole-Slide Imagescited by 24
  9. 2024Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Imagescited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Modal Tumor Survival Prediction via Graph-Guided Mixture of Experts
  2. 2026HistoMet: A Pan-Cancer Deep Learning Framework for Prognostic Prediction of Metastatic Progression and Site Tropism from Primary Tumor Histopathology
  3. 2026Anchor-driven Multi-modal Multi-scale Expert Selection for Survival Prediction
  4. 2024MMFusion: Multi-modality Diffusion Model for Lymph Node Metastasis Diagnosis in Esophageal Cancer
  5. 2023Gene-MOE: A sparsely gated prognosis and classification framework exploiting pan-cancer genomic information
  6. 2026SAGE: Semantic Explainability of Attention-Based Survival Models in Computational Pathology
  7. 2025Multi-Modal Mamba Modeling for Survival Prediction (M4Survive): Adapting Joint Foundation Model Representations
  8. 2025GraphMMP: A Graph Neural Network Model with Mutual Information and Global Fusion for Multimodal Medical Prognosis