# Generating crossmodal gene expression from cancer histopathology improves multimodal AI predictions

arXiv: https://arxiv.org/abs/2502.00568 · HTML: https://mlsanity.com/paper/2502.00568/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 1 Feb 2025.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Cross-Entropy Loss Functions: Theoretical Analysis and Applications](https://mlsanity.com/paper/2304.07288/) (2023)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Deep Learning for Identifying Metastatic Breast Cancer](https://mlsanity.com/paper/1606.05718/) (2016)
- [Transcriptome-wide prediction of prostate cancer gene expression from histopathology images using co-expression based convolutional neural networks](https://mlsanity.com/paper/2104.09310/) (2021)

## Built on by (most cited first, 5 of 5)

- [Conformal uncertainty quantification to evaluate predictive fairness of foundation AI model for skin lesion classes across patient demographics](https://mlsanity.com/paper/2503.23819/) (2025)
- [Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis](https://mlsanity.com/paper/2606.17115/) (2026)
- [A Multi-Cohort Validation of Censoring-Aware Conformal Lower Predictive Bounds for Pathology Survival Models](https://mlsanity.com/paper/2608.04025/) (2026)
- [Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classification](https://mlsanity.com/paper/2609.13303/) (2026)
- [Transcriptome-informed multi-modal AI for predicting neoadjuvant therapy response from breast cancer biopsies](https://mlsanity.com/paper/2610.03693/) (2026)

## Abstract (arXiv)

Emerging research has highlighted that artificial intelligence-based multimodal fusion of digital pathology and transcriptomic features can improve cancer diagnosis (grading/subtyping) and prognosis (survival risk) prediction. However, such direct fusion is impractical in clinical settings, where histopathology remains the gold standard and transcriptomic tests are rarely requested in public healthcare. We experiment on two publicly available multimodal datasets, The Cancer Genomic Atlas and the Clinical Proteomic Tumor Analysis Consortium, spanning four independent cohorts: glioma-glioblastoma, renal, uterine, and breast, and observe significant performance gains in gradation and risk estimation (p-value<0.05) when incorporating synthesized transcriptomic data with WSIs. Also, predictions using synthesized features were statistically close to those obtained with real transcriptomic data (p-value>0.05), consistently across cohorts. Here we show that with our diffusion based crossmodal generative AI model, PathGen, gene expressions synthesized from digital histopathology jointly predict cancer grading and patient survival risk with high accuracy (state-of-the-art performance), certainty (through conformal coverage guarantee) and interpretability (through distributed co-attention maps). PathGen code is available for open use on GitHub at https://github.com/Samiran-Dey/PathGen.