# TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction

arXiv: https://arxiv.org/abs/2209.05036 · HTML: https://mlsanity.com/paper/2209.05036/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [An Ensemble Approach for Patient Prognosis of Head and Neck Tumor Using Multimodal Data](https://mlsanity.com/paper/2202.12537/) (2022)
- [Deep Convolutional Neural Networks for Imaging Data Based Survival Analysis of Rectal Cancer](https://mlsanity.com/paper/1901.01449/) (2019)
- [AMINN: Autoencoder-based Multiple Instance Neural Network Improves Outcome Prediction of Multifocal Liver Metastases](https://mlsanity.com/paper/2012.06875/) (2020)
- [Automatic Segmentation of Head and Neck Tumor: How Powerful Transformers Are?](https://mlsanity.com/paper/2201.06251/) (2022)
- [M2Net: Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients](https://mlsanity.com/paper/2006.10135/) (2020)

## Built on by (most cited first, 18 of 18)

- [SurvRNC: Learning Ordered Representations for Survival Prediction using Rank-N-Contrast](https://mlsanity.com/paper/2403.10603/) (2024)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer](https://mlsanity.com/paper/2307.03427/) (2023)
- [H-DenseFormer: An Efficient Hybrid Densely Connected Transformer for Multimodal Tumor Segmentation](https://mlsanity.com/paper/2307.01486/) (2023)
- [A Multimodal and Multi-centric Head and Neck Cancer Dataset for Segmentation, Diagnosis and Outcome Prediction](https://mlsanity.com/paper/2509.00367/) (2025)
- [AdaMSS: Adaptive Multi-Modality Segmentation-to-Survival Learning for Survival Outcome Prediction from PET/CT Images](https://mlsanity.com/paper/2305.09946/) (2023)
- [Multiscale Progressive Text Prompt Network for Medical Image Segmentation](https://mlsanity.com/paper/2307.00174/) (2023)
- [PEMMA: Parameter-Efficient Multi-Modal Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/2404.13704/) (2024)
- [Survival Prediction in Lung Cancer through Multi-Modal Representation Learning](https://mlsanity.com/paper/2409.20179/) (2024)
- [Improved Prognostic Prediction of Pancreatic Cancer Using Multi-Phase CT by Integrating Neural Distance and Texture-Aware Transformer](https://mlsanity.com/paper/2308.00507/) (2023)
- [Diagnosis and Prognosis of Head and Neck Cancer Patients using Artificial Intelligence](https://mlsanity.com/paper/2306.00034/) (2023)
- [Radiomics-Informed Deep Learning for Classification of Atrial Fibrillation Sub-Types from Left-Atrium CT Volumes](https://mlsanity.com/paper/2308.06933/) (2023)
- [Joint Explicit and Implicit Cross-Modal Interaction Network for Anterior Chamber Inflammation Diagnosis](https://mlsanity.com/paper/2312.06171/) (2023)
- [Y-CA-Net: A Convolutional Attention Based Network for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2410.01003/) (2024)
- [Efficient Parameter Adaptation for Multi-Modal Medical Image Segmentation and Prognosis](https://mlsanity.com/paper/2504.13645/) (2025)
- [RobSurv: Vector Quantization-Based Multi-Modal Learning for Robust Cancer Survival Prediction](https://mlsanity.com/paper/2505.02529/) (2025)
- [USCNet: Transformer-Based Multimodal Fusion with Segmentation Guidance for Urolithiasis Classification](https://mlsanity.com/paper/2604.07141/) (2026)
- [Multimodal Assessment of Pancreatic Cancer Resectability Using Deep Learning](https://mlsanity.com/paper/2607.13826/) (2026)

## Abstract (arXiv)

When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient survival. We propose TMSS, an end-to-end Transformer based Multimodal network for Segmentation and Survival prediction that leverages the superiority of transformers that lies in their abilities to handle different modalities. The model was trained and validated for segmentation and prognosis tasks on the training dataset from the HEad & NeCK TumOR segmentation and the outcome prediction in PET/CT images challenge (HECKTOR). We show that the proposed prognostic model significantly outperforms state-of-the-art methods with a concordance index of 0.763+/-0.14 while achieving a comparable dice score of 0.772+/-0.030 to a standalone segmentation model. The code is publicly available.