# Path-GPTOmic: A Balanced Multi-modal Learning Framework for Survival Outcome Prediction

arXiv: https://arxiv.org/abs/2403.11375 · HTML: https://mlsanity.com/paper/2403.11375/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Smoothing the Disentangled Latent Style Space for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2106.09016/) (2021)
- [Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction](https://mlsanity.com/paper/2307.11952/) (2023)
- [MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning](https://mlsanity.com/paper/2303.05338/) (2023)
- [Latent Space Smoothing for Individually Fair Representations](https://mlsanity.com/paper/2111.13650/) (2021)
- [Unlabeled Data Guided Semi-supervised Histopathology Image Segmentation](https://mlsanity.com/paper/2012.09373/) (2020)
- [Smooth image-to-image translations with latent space interpolations](https://mlsanity.com/paper/2210.00841/) (2022)

## Built on by (most cited first, 5 of 5)

- [From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research](https://mlsanity.com/paper/2507.09028/) (2025)
- [Self-Normalizing Foundation Model for Enhanced Multi-Omics Data Analysis in Oncology](https://mlsanity.com/paper/2405.08226/) (2024)
- [White Light Specular Reflection Data Augmentation for Deep Learning Polyp Detection](https://mlsanity.com/paper/2505.05248/) (2025)
- [Adapting a Segmentation Foundation Model for Medical Image Classification](https://mlsanity.com/paper/2505.06217/) (2025)
- [PRISM: A Framework Harnessing Unsupervised Visual Representations and Textual Prompts for Explainable MACE Survival Prediction from Cardiac Cine MRI](https://mlsanity.com/paper/2508.19325/) (2025)

## Abstract (arXiv)

For predicting cancer survival outcomes, standard approaches in clinical research are often based on two main modalities: pathology images for observing cell morphology features, and genomic (e.g., bulk RNA-seq) for quantifying gene expressions. However, existing pathology-genomic multi-modal algorithms face significant challenges: (1) Valuable biological insights regarding genes and gene-gene interactions are frequently overlooked; (2) one modality often dominates the optimization process, causing inadequate training for the other modality. In this paper, we introduce a new multi-modal ``Path-GPTOmic" framework for cancer survival outcome prediction. First, to extract valuable biological insights, we regulate the embedding space of a foundation model, scGPT, initially trained on single-cell RNA-seq data, making it adaptable for bulk RNA-seq data. Second, to address the imbalance-between-modalities problem, we propose a gradient modulation mechanism tailored to the Cox partial likelihood loss for survival prediction. The contributions of the modalities are dynamically monitored and adjusted during the training process, encouraging that both modalities are sufficiently trained. Evaluated on two TCGA(The Cancer Genome Atlas) datasets, our model achieves substantially improved survival prediction accuracy.