# Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning

arXiv: https://arxiv.org/abs/2406.13979 · HTML: https://mlsanity.com/paper/2406.13979/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Cross-Modal Translation and Alignment for Survival Analysis](https://mlsanity.com/paper/2309.12855/) (2023)
- [Multi-modal learning for predicting the genotype of glioma](https://mlsanity.com/paper/2203.10852/) (2022)
- [Multi-task Learning of Histology and Molecular Markers for Classifying Diffuse Glioma](https://mlsanity.com/paper/2303.14845/) (2023)
- [Mutual Contrastive Low-rank Learning to Disentangle Whole Slide Image Representations for Glioma Grading](https://mlsanity.com/paper/2203.04013/) (2022)

## Built on by (most cited first, 4 of 4)

- [Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization](https://mlsanity.com/paper/2508.16479/) (2025)
- [CLIP-IT: CLIP-based Pairing for Histology Images Classification](https://mlsanity.com/paper/2504.16181/) (2025)
- [Adaptive Spatial Transcriptomics Interpolation via Cross-modal Cross-slice Modeling](https://mlsanity.com/paper/2505.10729/) (2025)
- [HaDM-ST: Histology-Assisted Differential Modeling for Spatial Transcriptomics Generation](https://mlsanity.com/paper/2508.07225/) (2025)

## Abstract (arXiv)

Multi-modal learning plays a crucial role in cancer diagnosis and prognosis. Current deep learning based multi-modal approaches are often limited by their abilities to model the complex correlations between genomics and histology data, addressing the intrinsic complexity of tumour ecosystem where both tumour and microenvironment contribute to malignancy. We propose a biologically interpretative and robust multi-modal learning framework to efficiently integrate histology images and genomics by decomposing the feature subspace of histology images and genomics, reflecting distinct tumour and microenvironment features. To enhance cross-modal interactions, we design a knowledge-driven subspace fusion scheme, consisting of a cross-modal deformable attention module and a gene-guided consistency strategy. Additionally, in pursuit of dynamically optimizing the subspace knowledge, we further propose a novel gradient coordination learning strategy. Extensive experiments demonstrate the effectiveness of the proposed method, outperforming state-of-the-art techniques in three downstream tasks of glioma diagnosis, tumour grading, and survival analysis. Our code is available at https://github.com/helenypzhang/Subspace-Multimodal-Learning.