# Joint Modelling Histology and Molecular Markers for Cancer Classification

arXiv: https://arxiv.org/abs/2502.07979 · HTML: https://mlsanity.com/paper/2502.07979/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [The Emerging Trends of Multi-Label Learning](https://mlsanity.com/paper/2011.11197/) (2020)
- [A Comprehensive Review of Computer-aided Whole-slide Image Analysis: from Datasets to Feature Extraction, Segmentation, Classification, and Detection Approaches](https://mlsanity.com/paper/2102.10553/) (2021)
- [Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction](https://mlsanity.com/paper/2307.11952/) (2023)
- [DisC-Diff: Disentangled Conditional Diffusion Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2303.13933/) (2023)
- [Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture](https://mlsanity.com/paper/1909.10726/) (2019)
- [Multi-modal learning for predicting the genotype of glioma](https://mlsanity.com/paper/2203.10852/) (2022)
- [ScanNet: A Fast and Dense Scanning Framework for Metastatic Breast Cancer Detection from Whole-Slide Images](https://mlsanity.com/paper/1707.09597/) (2017)
- [Multi-task Learning of Histology and Molecular Markers for Classifying Diffuse Glioma](https://mlsanity.com/paper/2303.14845/) (2023)
- [Phy-Diff: Physics-guided Hourglass Diffusion Model for Diffusion MRI Synthesis](https://mlsanity.com/paper/2406.03002/) (2024)
- [Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging](https://mlsanity.com/paper/2303.13610/) (2023)
- [Cross-modal Diffusion Modelling for Super-resolved Spatial Transcriptomics](https://mlsanity.com/paper/2404.12973/) (2024)
- [Combining datasets to increase the number of samples and improve model fitting](https://mlsanity.com/paper/2210.05165/) (2022)

## Abstract (arXiv)

Cancers are characterized by remarkable heterogeneity and diverse prognosis. Accurate cancer classification is essential for patient stratification and clinical decision-making. Although digital pathology has been advancing cancer diagnosis and prognosis, the paradigm in cancer pathology has shifted from purely relying on histology features to incorporating molecular markers. There is an urgent need for digital pathology methods to meet the needs of the new paradigm. We introduce a novel digital pathology approach to jointly predict molecular markers and histology features and model their interactions for cancer classification. Firstly, to mitigate the challenge of cross-magnification information propagation, we propose a multi-scale disentangling module, enabling the extraction of multi-scale features from high-magnification (cellular-level) to low-magnification (tissue-level) whole slide images. Further, based on the multi-scale features, we propose an attention-based hierarchical multi-task multi-instance learning framework to simultaneously predict histology and molecular markers. Moreover, we propose a co-occurrence probability-based label correlation graph network to model the co-occurrence of molecular markers. Lastly, we design a cross-modal interaction module with the dynamic confidence constrain loss and a cross-modal gradient modulation strategy, to model the interactions of histology and molecular markers. Our experiments demonstrate that our method outperforms other state-of-the-art methods in classifying glioma, histology features and molecular markers. Our method promises to promote precise oncology with the potential to advance biomedical research and clinical applications. The code is available at https://github.com/LHY1007/M3C2