# Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability

arXiv: https://arxiv.org/abs/2406.12897 · HTML: https://mlsanity.com/paper/2406.12897/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 103 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images](https://mlsanity.com/paper/2111.04740/) (2021)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [PathLDM: Text conditioned Latent Diffusion Model for Histopathology](https://mlsanity.com/paper/2309.00748/) (2023)
- [HookNet: multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images](https://mlsanity.com/paper/2006.12230/) (2020)
- [HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction](https://mlsanity.com/paper/2403.05396/) (2024)
- [LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity](https://mlsanity.com/paper/2404.03214/) (2024)
- [LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models](https://mlsanity.com/paper/2404.03118/) (2024)
- [Towards Explainable Graph Representations in Digital Pathology](https://mlsanity.com/paper/2007.00311/) (2020)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks](https://mlsanity.com/paper/2402.12525/) (2024)
- [Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Tasks](https://mlsanity.com/paper/2308.09033/) (2023)
- [On Image Search in Histopathology](https://mlsanity.com/paper/2401.08699/) (2024)
- [Deep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions](https://mlsanity.com/paper/2304.06662/) (2023)
- [A Multimodal Graph Neural Network Framework of Cancer Molecular Subtype Classification](https://mlsanity.com/paper/2302.12838/) (2023)
- [CLARA: Clinical Report Auto-completion](https://mlsanity.com/paper/2002.11701/) (2020)
- [Mini-DDSM: Mammography-based Automatic Age Estimation](https://mlsanity.com/paper/2010.00494/) (2020)
- [Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review](https://mlsanity.com/paper/2306.01546/) (2023)
- [Multi-modality fusion using canonical correlation analysis methods: Application in breast cancer survival prediction from histology and genomics](https://mlsanity.com/paper/2111.13987/) (2021)
- [Improving mitosis detection on histopathology images using large vision-language models](https://mlsanity.com/paper/2310.07176/) (2023)
- [MM-SurvNet: Deep Learning-Based Survival Risk Stratification in Breast Cancer Through Multimodal Data Fusion](https://mlsanity.com/paper/2402.11788/) (2024)
- [Envisioning MedCLIP: A Deep Dive into Explainability for Medical Vision-Language Models](https://mlsanity.com/paper/2403.18996/) (2024)
- [The Ability of Image-Language Explainable Models to Resemble Domain Expertise](https://mlsanity.com/paper/2209.09310/) (2022)

## Abstract (arXiv)

It is imperative that breast cancer is detected precisely and timely to improve patient outcomes. Diagnostic methodologies have traditionally relied on unimodal approaches; however, medical data analytics is integrating diverse data sources beyond conventional imaging. Using multi-modal techniques, integrating both image and non-image data, marks a transformative advancement in breast cancer diagnosis. The purpose of this review is to explore the burgeoning field of multimodal techniques, particularly the fusion of histopathology images with non-image data. Further, Explainable AI (XAI) will be used to elucidate the decision-making processes of complex algorithms, emphasizing the necessity of explainability in diagnostic processes. This review utilizes multi-modal data and emphasizes explainability to enhance diagnostic accuracy, clinician confidence, and patient engagement, ultimately fostering more personalized treatment strategies for breast cancer, while also identifying research gaps in multi-modality and explainability, guiding future studies, and contributing to the strategic direction of the field.