# MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features

arXiv: https://arxiv.org/abs/2311.11659 · HTML: https://mlsanity.com/paper/2311.11659/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network](https://mlsanity.com/paper/1606.00931/) (2016)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Time-to-Event Prediction with Neural Networks and Cox Regression](https://mlsanity.com/paper/1907.00825/) (2019)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks](https://mlsanity.com/paper/2107.13048/) (2021)
- [Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction](https://mlsanity.com/paper/2306.08330/) (2023)
- [Differentiable Zooming for Multiple Instance Learning on Whole-Slide Images](https://mlsanity.com/paper/2204.12454/) (2022)
- [Interpretable Survival Prediction for Colorectal Cancer using Deep Learning](https://mlsanity.com/paper/2011.08965/) (2020)
- [Multiple Instance Learning with Mixed Supervision in Gleason Grading](https://mlsanity.com/paper/2206.12798/) (2022)
- [Kernel Attention Transformer (KAT) for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2206.13156/) (2022)

## Built on by (most cited first, 8 of 8)

- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Unleashing the Infinity Power of Geometry: A Novel Geometry-Aware Transformer (GOAT) for Whole Slide Histopathology Image Analysis](https://mlsanity.com/paper/2402.05373/) (2024)
- [MurreNet: Modeling Holistic Multimodal Interactions Between Histopathology and Genomic Profiles for Survival Prediction](https://mlsanity.com/paper/2507.04891/) (2025)
- [PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction](https://mlsanity.com/paper/2509.20022/) (2025)
- [MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological images](https://mlsanity.com/paper/2410.15881/) (2024)
- [Foundation Models for Slide-level Cancer Subtyping in Digital Pathology](https://mlsanity.com/paper/2410.15886/) (2024)
- [A Hierarchical Geometry-guided Transformer for Histological Subtyping of Primary Liver Cancer](https://mlsanity.com/paper/2510.05657/) (2025)
- [Synergistic Information Disentanglement for Omni-modal Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2609.02118/) (2026)

## Abstract (arXiv)

The rapidly emerging field of deep learning-based computational pathology has shown promising results in utilizing whole slide images (WSIs) to objectively prognosticate cancer patients. However, most prognostic methods are currently limited to either histopathology or genomics alone, which inevitably reduces their potential to accurately predict patient prognosis. Whereas integrating WSIs and genomic features presents three main challenges: (1) the enormous heterogeneity of gigapixel WSIs which can reach sizes as large as 150,000x150,000 pixels; (2) the absence of a spatially corresponding relationship between histopathology images and genomic molecular data; and (3) the existing early, late, and intermediate multimodal feature fusion strategies struggle to capture the explicit interactions between WSIs and genomics. To ameliorate these issues, we propose the Mutual-Guided Cross-Modality Transformer (MGCT), a weakly-supervised, attention-based multimodal learning framework that can combine histology features and genomic features to model the genotype-phenotype interactions within the tumor microenvironment. To validate the effectiveness of MGCT, we conduct experiments using nearly 3,600 gigapixel WSIs across five different cancer types sourced from The Cancer Genome Atlas (TCGA). Extensive experimental results consistently emphasize that MGCT outperforms the state-of-the-art (SOTA) methods.