# Survival Prediction in Lung Cancer through Multi-Modal Representation Learning

arXiv: https://arxiv.org/abs/2409.20179 · HTML: https://mlsanity.com/paper/2409.20179/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction](https://mlsanity.com/paper/2307.11952/) (2023)
- [TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction](https://mlsanity.com/paper/2209.05036/) (2022)
- [An Ensemble Approach for Patient Prognosis of Head and Neck Tumor Using Multimodal Data](https://mlsanity.com/paper/2202.12537/) (2022)
- [Image-based Survival Analysis for Lung Cancer Patients using CNNs](https://mlsanity.com/paper/1808.09679/) (2018)
- [DeepMTS: Deep Multi-task Learning for Survival Prediction in Patients with Advanced Nasopharyngeal Carcinoma using Pretreatment PET/CT](https://mlsanity.com/paper/2109.07711/) (2021)
- [Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer](https://mlsanity.com/paper/2307.03427/) (2023)

## Built on by (most cited first, 2 of 2)

- [RobSurv: Vector Quantization-Based Multi-Modal Learning for Robust Cancer Survival Prediction](https://mlsanity.com/paper/2505.02529/) (2025)
- [Discovering Subgroups with Exceptional Survival Characteristics](https://mlsanity.com/paper/2602.22179/) (2026)

## Abstract (arXiv)

Survival prediction is a crucial task associated with cancer diagnosis and treatment planning. This paper presents a novel approach to survival prediction by harnessing comprehensive information from CT and PET scans, along with associated Genomic data. Current methods rely on either a single modality or the integration of multiple modalities for prediction without adequately addressing associations across patients or modalities. We aim to develop a robust predictive model for survival outcomes by integrating multi-modal imaging data with genetic information while accounting for associations across patients and modalities. We learn representations for each modality via a self-supervised module and harness the semantic similarities across the patients to ensure the embeddings are aligned closely. However, optimizing solely for global relevance is inadequate, as many pairs sharing similar high-level semantics, such as tumor type, are inadvertently pushed apart in the embedding space. To address this issue, we use a cross-patient module (CPM) designed to harness inter-subject correspondences. The CPM module aims to bring together embeddings from patients with similar disease characteristics. Our experimental evaluation of the dataset of Non-Small Cell Lung Cancer (NSCLC) patients demonstrates the effectiveness of our approach in predicting survival outcomes, outperforming state-of-the-art methods.