# RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy

arXiv: https://arxiv.org/abs/2204.12423 · HTML: https://mlsanity.com/paper/2204.12423/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 54 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [AIforCOVID: predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays. An Italian multicentre study](https://mlsanity.com/paper/2012.06531/) (2020)
- [DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis](https://mlsanity.com/paper/2106.06744/) (2021)

## Built on by (most cited first, 10 of 10)

- [Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review](https://mlsanity.com/paper/2303.06471/) (2023)
- [Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications](https://mlsanity.com/paper/2411.03782/) (2024)
- [Class Balancing Diversity Multimodal Ensemble for Alzheimer's Disease Diagnosis and Early Detection](https://mlsanity.com/paper/2410.10374/) (2024)
- [Can Foundation Models Really Segment Tumors? A Benchmarking Odyssey in Lung CT Imaging](https://mlsanity.com/paper/2505.01239/) (2025)
- [Deep Learning for Human Locomotion Analysis in Lower-Limb Exoskeletons: A Comparative Study](https://mlsanity.com/paper/2503.16904/) (2025)
- [Context-Gated Cross-Modal Perception with Visual Mamba for PET-CT Lung Tumor Segmentation](https://mlsanity.com/paper/2510.27508/) (2025)
- [Probabilistic NDVI Forecasting from Sparse Satellite Time Series and Weather Covariates](https://mlsanity.com/paper/2602.17683/) (2026)
- [MATNet: Multi-Level Fusion Transformer-Based Model for Day-Ahead PV Generation Forecasting](https://mlsanity.com/paper/2306.10356/) (2023)
- [Hybrid Quantum Neural Network for Multivariate Clinical Time Series Forecasting](https://mlsanity.com/paper/2603.08072/) (2026)
- [Longitudinal NSCLC Treatment Progression via Multimodal Generative Models](https://mlsanity.com/paper/2603.06147/) (2026)

## Abstract (arXiv)

The current cancer treatment practice collects multimodal data, such as radiology images, histopathology slides, genomics and clinical data. The importance of these data sources taken individually has fostered the recent raise of radiomics and pathomics, i.e. the extraction of quantitative features from radiology and histopathology images routinely collected to predict clinical outcomes or to guide clinical decisions using artificial intelligence algorithms. Nevertheless, how to combine them into a single multimodal framework is still an open issue. In this work we therefore develop a multimodal late fusion approach that combines hand-crafted features computed from radiomics, pathomics and clinical data to predict radiation therapy treatment outcomes for non-small-cell lung cancer patients. Within this context, we investigate eight different late fusion rules (i.e. product, maximum, minimum, mean, decision template, Dempster-Shafer, majority voting, and confidence rule) and two patient-wise aggregation rules leveraging the richness of information given by computer tomography images and whole-slide scans. The experiments in leave-one-patient-out cross-validation on an in-house cohort of 33 patients show that the proposed multimodal paradigm with an AUC equal to 90.9% outperforms each unimodal approach, suggesting that data integration can advance precision medicine. As a further contribution, we also compare the hand-crafted representations with features automatically computed by deep networks, and the late fusion paradigm with early fusion, another popular multimodal approach. In both cases, the experiments show that the proposed multimodal approach provides the best results.