# VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction

arXiv: https://arxiv.org/abs/2503.19367 · HTML: https://mlsanity.com/paper/2503.19367/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction](https://mlsanity.com/paper/2306.08330/) (2023)
- [Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11643/) (2024)
- [Cross-Modal Translation and Alignment for Survival Analysis](https://mlsanity.com/paper/2309.12855/) (2023)
- [Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology](https://mlsanity.com/paper/2402.17228/) (2024)
- [Multimodal Prototyping for cancer survival prediction](https://mlsanity.com/paper/2407.00224/) (2024)
- [Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction](https://mlsanity.com/paper/2401.01646/) (2024)
- [WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering](https://mlsanity.com/paper/2407.05603/) (2024)
- [MoME: Mixture of Multimodal Experts for Cancer Survival Prediction](https://mlsanity.com/paper/2406.09696/) (2024)
- [Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Images](https://mlsanity.com/paper/2403.10040/) (2024)

## Abstract (arXiv)

Multimodal learning combining pathology images and genomic sequences enhances cancer survival analysis but faces clinical implementation barriers due to limited access to genomic sequencing in under-resourced regions. To enable survival prediction using only whole-slide images (WSI), we propose the Visual-Genomic Answering-Guided Transformer (VGAT), a framework integrating Visual Question Answering (VQA) techniques for genomic modality reconstruction. By adapting VQA's text feature extraction approach, we derive stable genomic representations that circumvent dimensionality challenges in raw genomic data. Simultaneously, a cluster-based visual prompt module selectively enhances discriminative WSI patches, addressing noise from unfiltered image regions. Evaluated across five TCGA datasets, VGAT outperforms existing WSI-only methods, demonstrating the viability of genomic-informed inference without sequencing. This approach bridges multimodal research and clinical feasibility in resource-constrained settings. The code link is https://github.com/CZZZZZZZZZZZZZZZZZ/VGAT.