# Multi-modality fusion using canonical correlation analysis methods: Application in breast cancer survival prediction from histology and genomics

arXiv: https://arxiv.org/abs/2111.13987 · HTML: https://mlsanity.com/paper/2111.13987/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations](https://mlsanity.com/paper/2109.14910/) (2021)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [Brain tumor segmentation with missing modalities via latent multi-source correlation representation](https://mlsanity.com/paper/2003.08870/) (2020)
- [Multimodal fusion of imaging and genomics for lung cancer recurrence prediction](https://mlsanity.com/paper/2002.01982/) (2020)
- [Collaborative Regression](https://mlsanity.com/paper/1401.5823/) (2014)
- [G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification](https://mlsanity.com/paper/2101.11656/) (2021)
- [Multimodal fusion using sparse CCA for breast cancer survival prediction](https://mlsanity.com/paper/2103.05432/) (2021)
- [Correlating Cellular Features with Gene Expression using CCA](https://mlsanity.com/paper/1802.08910/) (2018)

## Built on by (most cited first, 4 of 4)

- [Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data](https://mlsanity.com/paper/2302.06232/) (2023)
- [MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction](https://mlsanity.com/paper/2307.07093/) (2023)
- [MM-SurvNet: Deep Learning-Based Survival Risk Stratification in Breast Cancer Through Multimodal Data Fusion](https://mlsanity.com/paper/2402.11788/) (2024)
- [Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability](https://mlsanity.com/paper/2406.12897/) (2024)

## Abstract (arXiv)

The availability of multi-modality datasets provides a unique opportunity to characterize the same object of interest using multiple viewpoints more comprehensively. In this work, we investigate the use of canonical correlation analysis (CCA) and penalized variants of CCA (pCCA) for the fusion of two modalities. We study a simple graphical model for the generation of two-modality data. We analytically show that, with known model parameters, posterior mean estimators that jointly use both modalities outperform arbitrary linear mixing of single modality posterior estimators in latent variable prediction. Penalized extensions of CCA (pCCA) that incorporate domain knowledge can discover correlations with high-dimensional, low-sample data, whereas traditional CCA is inapplicable. To facilitate the generation of multi-dimensional embeddings with pCCA, we propose two matrix deflation schemes that enforce desirable properties exhibited by CCA. We propose a two-stage prediction pipeline using pCCA embeddings generated with deflation for latent variable prediction by combining all the above. On simulated data, our proposed model drastically reduces the mean-squared error in latent variable prediction. When applied to publicly available histopathology data and RNA-sequencing data from The Cancer Genome Atlas (TCGA) breast cancer patients, our model can outperform principal components analysis (PCA) embeddings of the same dimension in survival prediction.