# SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival

arXiv: https://arxiv.org/abs/2403.09290 · HTML: https://mlsanity.com/paper/2403.09290/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis](https://mlsanity.com/paper/2211.09117/) (2022)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Contrastive Masked Autoencoders are Stronger Vision Learners](https://mlsanity.com/paper/2207.13532/) (2022)
- [ConvMAE: Masked Convolution Meets Masked Autoencoders](https://mlsanity.com/paper/2205.03892/) (2022)
- [Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction](https://mlsanity.com/paper/2304.06819/) (2023)
- [Fine-Tuning and Training of DenseNet for Histopathology Image Representation Using TCGA Diagnostic Slides](https://mlsanity.com/paper/2101.07903/) (2021)
- [Heterogeneous Graph Masked Autoencoders](https://mlsanity.com/paper/2208.09957/) (2022)
- [Reduce Information Loss in Transformers for Pluralistic Image Inpainting](https://mlsanity.com/paper/2205.05076/) (2022)
- [Masked Autoencoders Enable Efficient Knowledge Distillers](https://mlsanity.com/paper/2208.12256/) (2022)
- [Learning and Reasoning with the Graph Structure Representation in Robotic Surgery](https://mlsanity.com/paper/2007.03357/) (2020)
- [Pooling Regularized Graph Neural Network for fMRI Biomarker Analysis](https://mlsanity.com/paper/2007.14589/) (2020)
- [Noise-reducing attention cross fusion learning transformer for histological image classification of osteosarcoma](https://mlsanity.com/paper/2204.13838/) (2022)
- [LDCSF: Local depth convolution-based Swim framework for classifying multi-label histopathology images](https://mlsanity.com/paper/2308.10446/) (2023)
- [PACS: Prediction and analysis of cancer subtypes from multi-omics data based on a multi-head attention mechanism model](https://mlsanity.com/paper/2308.10917/) (2023)

## Built on by (most cited first, 2 of 2)

- [Graph Neural Networks in Histopathology: Emerging Trends and Future Directions](https://mlsanity.com/paper/2406.12808/) (2024)
- [Opportunities and challenges in the application of large artificial intelligence models in radiology](https://mlsanity.com/paper/2403.16112/) (2024)

## Abstract (arXiv)

Accurately predicting the survival rate of cancer patients is crucial for aiding clinicians in planning appropriate treatment, reducing cancer-related medical expenses, and significantly enhancing patients' quality of life. Multimodal prediction of cancer patient survival offers a more comprehensive and precise approach. However, existing methods still grapple with challenges related to missing multimodal data and information interaction within modalities. This paper introduces SELECTOR, a heterogeneous graph-aware network based on convolutional mask encoders for robust multimodal prediction of cancer patient survival. SELECTOR comprises feature edge reconstruction, convolutional mask encoder, feature cross-fusion, and multimodal survival prediction modules. Initially, we construct a multimodal heterogeneous graph and employ the meta-path method for feature edge reconstruction, ensuring comprehensive incorporation of feature information from graph edges and effective embedding of nodes. To mitigate the impact of missing features within the modality on prediction accuracy, we devised a convolutional masked autoencoder (CMAE) to process the heterogeneous graph post-feature reconstruction. Subsequently, the feature cross-fusion module facilitates communication between modalities, ensuring that output features encompass all features of the modality and relevant information from other modalities. Extensive experiments and analysis on six cancer datasets from TCGA demonstrate that our method significantly outperforms state-of-the-art methods in both modality-missing and intra-modality information-confirmed cases. Our codes are made available at https://github.com/panliangrui/Selector.