# Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review

arXiv: https://arxiv.org/abs/2211.14847 · HTML: https://mlsanity.com/paper/2211.14847/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 113 references that are in the map.
- First posted on arXiv 27 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [A Fine-Grained Analysis on Distribution Shift](https://mlsanity.com/paper/2110.11328/) (2021)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Siloed Federated Learning for Multi-Centric Histopathology Datasets](https://mlsanity.com/paper/2008.07424/) (2020)
- [Streaming convolutional neural networks for end-to-end learning with multi-megapixel images](https://mlsanity.com/paper/1911.04432/) (2019)
- [SlideGraph+: Whole Slide Image Level Graphs to Predict HER2Status in Breast Cancer](https://mlsanity.com/paper/2110.06042/) (2021)
- [Learning domain-agnostic visual representation for computational pathology using medically-irrelevant style transfer augmentation](https://mlsanity.com/paper/2102.01678/) (2021)
- [Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers](https://mlsanity.com/paper/2109.05819/) (2021)
- [Predicting molecular phenotypes from histopathology images: a transcriptome-wide expression-morphology analysis in breast cancer](https://mlsanity.com/paper/2009.08917/) (2020)
- [ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification](https://mlsanity.com/paper/2202.07570/) (2022)
- [Detection of prostate cancer in whole-slide images through end-to-end training with image-level labels](https://mlsanity.com/paper/2006.03394/) (2020)
- [Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology](https://mlsanity.com/paper/1806.05083/) (2018)
- [High Resolution Medical Image Analysis with Spatial Partitioning](https://mlsanity.com/paper/1909.03108/) (2019)
- [HEROHE Challenge: assessing HER2 status in breast cancer without immunohistochemistry or in situ hybridization](https://mlsanity.com/paper/2111.04738/) (2021)
- [Failure Modes of Domain Generalization Algorithms](https://mlsanity.com/paper/2111.13733/) (2021)
- [H&E-based Computational Biomarker Enables Universal EGFR Screening for Lung Adenocarcinoma](https://mlsanity.com/paper/2206.10573/) (2022)
- [Deep Interactive Learning-based ovarian cancer segmentation of H&E-stained whole slide images to study morphological patterns of BRCA mutation](https://mlsanity.com/paper/2203.15015/) (2022)
- [Attention-based Interpretable Regression of Gene Expression in Histology](https://mlsanity.com/paper/2208.13776/) (2022)
- [Rethinking Machine Learning Model Evaluation in Pathology](https://mlsanity.com/paper/2204.05205/) (2022)
- [Deep Learning-based Frozen Section to FFPE Translation](https://mlsanity.com/paper/2107.11786/) (2021)
- [XDEEP-MSI: Explainable Bias-Rejecting Microsatellite Instability Deep Learning System In Colorectal Cancer](https://mlsanity.com/paper/2110.15350/) (2021)
- [Exploring Genetic-histologic Relationships in Breast Cancer](https://mlsanity.com/paper/2103.08082/) (2021)

## Built on by (most cited first, 2 of 2)

- [Towards Hierarchical Regional Transformer-based Multiple Instance Learning](https://mlsanity.com/paper/2308.12634/) (2023)
- [Cellpose+, a morphological analysis tool for feature extraction of stained cell images](https://mlsanity.com/paper/2410.18738/) (2024)

## Abstract (arXiv)

Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of viruses. This article reviews the diverse applications across cancer types and the methodology to train and validate these models on whole slide images. From bottom-up to pathologist-driven to hybrid approaches, the leading trends include a variety of weakly supervised deep learning-based approaches, as well as mechanisms for training strongly supervised models in select situations. While results of these algorithms look promising, some challenges still persist, including small training sets, rigorous validation, and model explainability. Biomarker prediction models may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible. They also create new opportunities in quantifying intratumoral heterogeneity and predicting patient outcomes.