# Cross-Modal Translation and Alignment for Survival Analysis

arXiv: https://arxiv.org/abs/2309.12855 · HTML: https://mlsanity.com/paper/2309.12855/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Machine Learning for Survival Analysis: A Survey](https://mlsanity.com/paper/1708.04649/) (2017)
- [Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primary](https://mlsanity.com/paper/2006.13932/) (2020)
- [Deep learning-based survival prediction for multiple cancer types using histopathology images](https://mlsanity.com/paper/1912.07354/) (2019)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [Transformer-based Personalized Attention Mechanism for Medical Images with Clinical Records](https://mlsanity.com/paper/2206.03003/) (2022)

## Built on by (most cited first, 30 of 70)

- [Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology](https://mlsanity.com/paper/2402.17228/) (2024)
- [A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model](https://mlsanity.com/paper/2407.15362/) (2024)
- [SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding](https://mlsanity.com/paper/2410.11761/) (2024)
- [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11618/) (2024)
- [Multimodal Prototyping for cancer survival prediction](https://mlsanity.com/paper/2407.00224/) (2024)
- [Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction](https://mlsanity.com/paper/2401.01646/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [PathBench: A comprehensive comparison benchmark for pathology foundation models towards precision oncology](https://mlsanity.com/paper/2505.20202/) (2025)
- [MoME: Mixture of Multimodal Experts for Cancer Survival Prediction](https://mlsanity.com/paper/2406.09696/) (2024)
- [Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction](https://mlsanity.com/paper/2411.14001/) (2024)
- [FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2411.14743/) (2024)
- [PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning](https://mlsanity.com/paper/2403.08967/) (2024)
- [Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology](https://mlsanity.com/paper/2409.09369/) (2024)
- [Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis](https://mlsanity.com/paper/2404.02394/) (2024)
- [Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncoder](https://mlsanity.com/paper/2503.09496/) (2025)
- [Distilled Prompt Learning for Incomplete Multimodal Survival Prediction](https://mlsanity.com/paper/2503.01653/) (2025)
- [Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance](https://mlsanity.com/paper/2505.11997/) (2025)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [SurvMamba: State Space Model with Multi-grained Multi-modal Interaction for Survival Prediction](https://mlsanity.com/paper/2404.08027/) (2024)
- [GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation](https://mlsanity.com/paper/2407.05540/) (2024)
- [Histo-Genomic Knowledge Distillation For Cancer Prognosis From Histopathology Whole Slide Images](https://mlsanity.com/paper/2403.10040/) (2024)
- [Multimodal Cross-Task Interaction for Survival Analysis in Whole Slide Pathological Images](https://mlsanity.com/paper/2406.17225/) (2024)
- [Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations](https://mlsanity.com/paper/2407.17726/) (2024)
- [SurvAgent: Hierarchical CoT-Enhanced Case Banking and Dichotomy-Based Multi-Agent System for Multimodal Survival Prediction](https://mlsanity.com/paper/2511.16635/) (2025)
- [Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning](https://mlsanity.com/paper/2603.00667/) (2026)
- [Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning](https://mlsanity.com/paper/2406.13979/) (2024)
- [RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency](https://mlsanity.com/paper/2411.15076/) (2024)
- [Memory-Augmented Incomplete Multimodal Survival Prediction via Cross-Slide and Gene-Attentive Hypergraph Learning](https://mlsanity.com/paper/2506.19324/) (2025)
- [AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles](https://mlsanity.com/paper/2412.16701/) (2024)
- [Prototype-Guided Cross-Modal Knowledge Enhancement for Adaptive Survival Prediction](https://mlsanity.com/paper/2503.10726/) (2025)

## Abstract (arXiv)

With the rapid advances in high-throughput sequencing technologies, the focus of survival analysis has shifted from examining clinical indicators to incorporating genomic profiles with pathological images. However, existing methods either directly adopt a straightforward fusion of pathological features and genomic profiles for survival prediction, or take genomic profiles as guidance to integrate the features of pathological images. The former would overlook intrinsic cross-modal correlations. The latter would discard pathological information irrelevant to gene expression. To address these issues, we present a Cross-Modal Translation and Alignment (CMTA) framework to explore the intrinsic cross-modal correlations and transfer potential complementary information. Specifically, we construct two parallel encoder-decoder structures for multi-modal data to integrate intra-modal information and generate cross-modal representation. Taking the generated cross-modal representation to enhance and recalibrate intra-modal representation can significantly improve its discrimination for comprehensive survival analysis. To explore the intrinsic crossmodal correlations, we further design a cross-modal attention module as the information bridge between different modalities to perform cross-modal interactions and transfer complementary information. Our extensive experiments on five public TCGA datasets demonstrate that our proposed framework outperforms the state-of-the-art methods.