# Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction

arXiv: https://arxiv.org/abs/2307.11952 · HTML: https://mlsanity.com/paper/2307.11952/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 31 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Graph Convolutional Networks for Multi-modality Medical Imaging: Methods, Architectures, and Clinical Applications](https://mlsanity.com/paper/2202.08916/) (2022)

## Built on by (most cited first, 30 of 32)

- [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11618/) (2024)
- [Multimodal Prototyping for cancer survival prediction](https://mlsanity.com/paper/2407.00224/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [MoME: Mixture of Multimodal Experts for Cancer Survival Prediction](https://mlsanity.com/paper/2406.09696/) (2024)
- [Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction](https://mlsanity.com/paper/2411.14001/) (2024)
- [PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning](https://mlsanity.com/paper/2403.08967/) (2024)
- [M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images](https://mlsanity.com/paper/2409.15092/) (2024)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images](https://mlsanity.com/paper/2401.10608/) (2024)
- [Cost-effective Instruction Learning for Pathology Vision and Language Analysis](https://mlsanity.com/paper/2407.17734/) (2024)
- [From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research](https://mlsanity.com/paper/2507.09028/) (2025)
- [Path-GPTOmic: A Balanced Multi-modal Learning Framework for Survival Outcome Prediction](https://mlsanity.com/paper/2403.11375/) (2024)
- [A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planning](https://mlsanity.com/paper/2506.07236/) (2025)
- [Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning](https://mlsanity.com/paper/2506.08854/) (2025)
- [PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction](https://mlsanity.com/paper/2509.20022/) (2025)
- [Genomics-guided Representation Learning for Pathologic Pan-cancer Tumor Microenvironment Subtype Prediction](https://mlsanity.com/paper/2406.06517/) (2024)
- [Survival Prediction in Lung Cancer through Multi-Modal Representation Learning](https://mlsanity.com/paper/2409.20179/) (2024)
- [PathoGen-X: A Cross-Modal Genomic Feature Trans-Align Network for Enhanced Survival Prediction from Histopathology Images](https://mlsanity.com/paper/2411.00749/) (2024)
- [ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathology](https://mlsanity.com/paper/2503.17564/) (2025)
- [Libra-MIL: Multimodal Prototypes Stereoscopic Infused with Task-specific Language Priors for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2511.07941/) (2025)
- [Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis](https://mlsanity.com/paper/2511.18089/) (2025)
- [Prototype-driven fusion of pathology and spatial transcriptomics for interpretable survival prediction](https://mlsanity.com/paper/2602.12441/) (2026)
- [A Multimodal Object-level Contrast Learning Method for Cancer Survival Risk Prediction](https://mlsanity.com/paper/2409.02145/) (2024)
- [Predicting Deterioration in Mild Cognitive Impairment with Survival Transformers, Extreme Gradient Boosting and Cox Proportional Hazard Modelling](https://mlsanity.com/paper/2409.16231/) (2024)
- [Continually Evolved Multimodal Foundation Models for Cancer Prognosis](https://mlsanity.com/paper/2501.18170/) (2025)
- [Joint Modelling Histology and Molecular Markers for Cancer Classification](https://mlsanity.com/paper/2502.07979/) (2025)
- [Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining](https://mlsanity.com/paper/2505.12711/) (2025)
- [Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis](https://mlsanity.com/paper/2512.01116/) (2025)
- [Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathology](https://mlsanity.com/paper/2602.13944/) (2026)

## Abstract (arXiv)

Survival outcome assessment is challenging and inherently associated with multiple clinical factors (e.g., imaging and genomics biomarkers) in cancer. Enabling multimodal analytics promises to reveal novel predictive patterns of patient outcomes. In this study, we propose a multimodal transformer (PathOmics) integrating pathology and genomics insights into colon-related cancer survival prediction. We emphasize the unsupervised pretraining to capture the intrinsic interaction between tissue microenvironments in gigapixel whole slide images (WSIs) and a wide range of genomics data (e.g., mRNA-sequence, copy number variant, and methylation). After the multimodal knowledge aggregation in pretraining, our task-specific model finetuning could expand the scope of data utility applicable to both multi- and single-modal data (e.g., image- or genomics-only). We evaluate our approach on both TCGA colon and rectum cancer cohorts, showing that the proposed approach is competitive and outperforms state-of-the-art studies. Finally, our approach is desirable to utilize the limited number of finetuned samples towards data-efficient analytics for survival outcome prediction. The code is available at https://github.com/Cassie07/PathOmics.