# Multimodal Deep Learning for Subtype Classification in Breast Cancer Using Histopathological Images and Gene Expression Data

arXiv: https://arxiv.org/abs/2503.02849 · HTML: https://mlsanity.com/paper/2503.02849/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 7 of 7)

- [Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders](https://mlsanity.com/paper/2104.11598/) (2021)
- [3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces](https://mlsanity.com/paper/2104.11532/) (2021)
- [Neural Speaker Embeddings for Ultrasound-based Silent Speech Interfaces](https://mlsanity.com/paper/2106.04552/) (2021)
- [Voice Activity Detection for Ultrasound-based Silent Speech Interfaces using Convolutional Neural Networks](https://mlsanity.com/paper/2105.13718/) (2021)
- [Adaptation of Tacotron2-based Text-To-Speech for Articulatory-to-Acoustic Mapping using Ultrasound Tongue Imaging](https://mlsanity.com/paper/2107.12051/) (2021)
- [Improved Processing of Ultrasound Tongue Videos by Combining ConvLSTM and 3D Convolutional Networks](https://mlsanity.com/paper/2206.12947/) (2022)
- [Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks](https://mlsanity.com/paper/2305.19130/) (2023)

## Abstract (arXiv)

Molecular subtyping of breast cancer is crucial for personalized treatment and prognosis. Traditional classification approaches rely on either histopathological images or gene expression profiling, limiting their predictive power. In this study, we propose a deep multimodal learning framework that integrates histopathological images and gene expression data to classify breast cancer into BRCA.Luminal and BRCA.Basal / Her2 subtypes. Our approach employs a ResNet-50 model for image feature extraction and fully connected layers for gene expression processing, with a cross-attention fusion mechanism to enhance modality interaction. We conduct extensive experiments using five-fold cross-validation, demonstrating that our multimodal integration outperforms unimodal approaches in terms of classification accuracy, precision-recall AUC, and F1-score. Our findings highlight the potential of deep learning for robust and interpretable breast cancer subtype classification, paving the way for improved clinical decision-making.