mlsanityData to 7 Oct 2026

Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data

arXiv 2107.00648 ↗First posted 1 Jul 2021cs.CV cs.LG cs.MMcited by 29

Not on the map. Why?

Abstract (arXiv)

Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables into a comprehensive multimodal risk score. Prognostic embeddings from each modality are learned and combined via attention-gated tensor fusion. To maximize the information gleaned from each modality, we introduce a multimodal orthogonalization (MMO) loss term that increases model performance by incentivizing constituent embeddings to be more complementary. DOF predicts OS in glioma patients with a median C-index of 0.788 +/- 0.067, significantly outperforming (p=0.023) the best performing unimodal model with a median C-index of 0.718 +/- 0.064. The prognostic model significantly stratifies glioma patients by OS within clinical subsets, adding further granularity to prognostic clinical grading and molecular subtyping.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 5 of its 29 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

5 of 29 references.

References · 5

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  3. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  4. 2017OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learningcited by 37
  5. 2020Multimodal fusion of imaging and genomics for lung cancer recurrence predictioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 4; 2024: 9; 2025: 4; 2026: 3.

29 papers.

Sort:

By area:

  1. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  2. 2022Causal Machine Learning for Healthcare and Precision Medicinecited by 68
  3. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
  4. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  5. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  6. 2022Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Datacited by 25
  7. 2023Medical Vision Language Pretraining: A surveycited by 24
  8. 2022Multimodal Hate Speech Detection from Bengali Memes and Textscited by 15
  9. 2024Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysiscited by 13
  10. 2024Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applicationscited by 13
  11. 2022Deep Multimodal Guidance for Medical Image Classificationcited by 11
  12. 2022RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapycited by 10
17 more
  1. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  2. 2024DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Datacited by 10
  3. 2021All You Need is Color: Image based Spatial Gene Expression Prediction using Neural Stain Learningcited by 7
  4. 2022Granger Causal Chain Discovery for Sepsis-Associated Derangements via Continuous-Time Hawkes Processescited by 6
  5. 2023Improved Multimodal Fusion for Small Datasets with Auxiliary Supervisioncited by 6
  6. 2023One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Datacited by 6
  7. 2021Multi-modality fusion using canonical correlation analysis methods: Application in breast cancer survival prediction from histology and genomicscited by 4
  8. 2024Completed Feature Disentanglement Learning for Multimodal MRIs Analysiscited by 4
  9. 2024M2Fusion: Bayesian-based Multimodal Multi-level Fusion on Colorectal Cancer Microsatellite Instability Predictioncited by 2
  10. 2024MOAB: Multi-Modal Outer Arithmetic Block For Fusion Of Histopathological Images And Genetic Data For Brain Tumor Gradingcited by 1
  11. 2025Multi-Modal Mamba Modeling for Survival Prediction (M4Survive): Adapting Joint Foundation Model Representationscited by 1
  12. 2025Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patientscited by 1
  13. 2025DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalitiescited by 1
  14. 2025Deep Learning-Based Computer Vision Models for Early Cancer Detection Using Multimodal Medical Imaging and Radiogenomic Integration Frameworkscited by 0
  15. 2026Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuningcited by 0
  16. 2026Trimodal Glioma Representation Alignment via Volumetric Contrastive Learningcited by 0
  17. 2026SAGEAgent: A Self-Evolving Agent for Cost-Aware Modality Acquisition in Multimodal Survival Predictioncited by 0