# MoME: Mixture of Multimodal Experts for Cancer Survival Prediction

arXiv: https://arxiv.org/abs/2406.09696 · HTML: https://mlsanity.com/paper/2406.09696/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction](https://mlsanity.com/paper/2306.08330/) (2023)
- [Cross-Modal Translation and Alignment for Survival Analysis](https://mlsanity.com/paper/2309.12855/) (2023)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction](https://mlsanity.com/paper/2401.01646/) (2024)
- [Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction](https://mlsanity.com/paper/2307.11952/) (2023)
- [Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2301.08125/) (2023)
- [TAKT: Target-Aware Knowledge Transfer for Whole Slide Image Classification](https://mlsanity.com/paper/2303.05780/) (2023)

## Built on by (most cited first, 23 of 23)

- [Recent Advances of Multimodal Continual Learning: A Comprehensive Survey](https://mlsanity.com/paper/2410.05352/) (2024)
- [Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction](https://mlsanity.com/paper/2411.14001/) (2024)
- [FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2411.14743/) (2024)
- [A Survey of Pathology Foundation Model: Progress and Future Directions](https://mlsanity.com/paper/2504.04045/) (2025)
- [Distilled Prompt Learning for Incomplete Multimodal Survival Prediction](https://mlsanity.com/paper/2503.01653/) (2025)
- [Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Prediction](https://mlsanity.com/paper/2509.18284/) (2025)
- [MurreNet: Modeling Holistic Multimodal Interactions Between Histopathology and Genomic Profiles for Survival Prediction](https://mlsanity.com/paper/2507.04891/) (2025)
- [CoC: Chain-of-Cancer based on Cross-Modal Autoregressive Traction for Survival Prediction](https://mlsanity.com/paper/2506.15696/) (2025)
- [Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization](https://mlsanity.com/paper/2508.16479/) (2025)
- [A Multimodal Foundation Model to Enhance Generalizability and Data Efficiency for Pan-cancer Prognosis Prediction](https://mlsanity.com/paper/2509.12600/) (2025)
- [ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis](https://mlsanity.com/paper/2509.16900/) (2025)
- [CLIP-IT: CLIP-based Pairing for Histology Images Classification](https://mlsanity.com/paper/2504.16181/) (2025)
- [Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification](https://mlsanity.com/paper/2505.15504/) (2025)
- [Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patients](https://mlsanity.com/paper/2507.06994/) (2025)
- [impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction](https://mlsanity.com/paper/2508.09195/) (2025)
- [Enhancing WSI-Based Survival Analysis with Report-Auxiliary Self-Distillation](https://mlsanity.com/paper/2509.15608/) (2025)
- [Multimodal Outer Arithmetic Block Dual Fusion of Whole Slide Images and Omics Data for Precision Oncology](https://mlsanity.com/paper/2411.17418/) (2024)
- [VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction](https://mlsanity.com/paper/2503.19367/) (2025)
- [Masked Omics Modeling for Multimodal Representation Learning across Histopathology and Molecular Profiles](https://mlsanity.com/paper/2508.00969/) (2025)
- [Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Prediction](https://mlsanity.com/paper/2510.06113/) (2025)
- [Advancing Cancer Prognosis with Hierarchical Fusion of Genomic, Proteomic and Pathology Imaging Data from a Systems Biology Perspective](https://mlsanity.com/paper/2603.13787/) (2026)
- [ChronoSurv: A Clinical Pathway-Guided Graph Framework for Multimodal Survival Analysis](https://mlsanity.com/paper/2606.19140/) (2026)
- [PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology](https://mlsanity.com/paper/2608.25970/) (2026)

## Abstract (arXiv)

Survival analysis, as a challenging task, requires integrating Whole Slide Images (WSIs) and genomic data for comprehensive decision-making. There are two main challenges in this task: significant heterogeneity and complex inter- and intra-modal interactions between the two modalities. Previous approaches utilize co-attention methods, which fuse features from both modalities only once after separate encoding. However, these approaches are insufficient for modeling the complex task due to the heterogeneous nature between the modalities. To address these issues, we propose a Biased Progressive Encoding (BPE) paradigm, performing encoding and fusion simultaneously. This paradigm uses one modality as a reference when encoding the other. It enables deep fusion of the modalities through multiple alternating iterations, progressively reducing the cross-modal disparities and facilitating complementary interactions. Besides modality heterogeneity, survival analysis involves various biomarkers from WSIs, genomics, and their combinations. The critical biomarkers may exist in different modalities under individual variations, necessitating flexible adaptation of the models to specific scenarios. Therefore, we further propose a Mixture of Multimodal Experts (MoME) layer to dynamically selects tailored experts in each stage of the BPE paradigm. Experts incorporate reference information from another modality to varying degrees, enabling a balanced or biased focus on different modalities during the encoding process. Extensive experimental results demonstrate the superior performance of our method on various datasets, including TCGA-BLCA, TCGA-UCEC and TCGA-LUAD. Codes are available at https://github.com/BearCleverProud/MoME.