# DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer

arXiv: https://arxiv.org/abs/2107.09405 · HTML: https://mlsanity.com/paper/2107.09405/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 20 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Neural Image Compression for Gigapixel Histopathology Image Analysis](https://mlsanity.com/paper/1811.02840/) (2018)
- [CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2101.06871/) (2021)
- [Streaming convolutional neural networks for end-to-end learning with multi-megapixel images](https://mlsanity.com/paper/1911.04432/) (2019)
- [Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding](https://mlsanity.com/paper/1910.10825/) (2019)
- [Detection of prostate cancer in whole-slide images through end-to-end training with image-level labels](https://mlsanity.com/paper/2006.03394/) (2020)
- [Extending Unsupervised Neural Image Compression With Supervised Multitask Learning](https://mlsanity.com/paper/2004.07041/) (2020)

## Built on by (most cited first, 27 of 27)

- [Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction](https://mlsanity.com/paper/2304.06819/) (2023)
- [Phikon-v2, A large and public feature extractor for biomarker prediction](https://mlsanity.com/paper/2409.09173/) (2024)
- [An Aggregation of Aggregation Methods in Computational Pathology](https://mlsanity.com/paper/2211.01256/) (2022)
- [Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directions](https://mlsanity.com/paper/2303.12484/) (2023)
- [Hierarchical discriminative learning improves visual representations of biomedical microscopy](https://mlsanity.com/paper/2303.01605/) (2023)
- [Incorporating intratumoral heterogeneity into weakly-supervised deep learning models via variance pooling](https://mlsanity.com/paper/2206.08885/) (2022)
- [Deep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions](https://mlsanity.com/paper/2304.06662/) (2023)
- [Benchmarking Embedding Aggregation Methods in Computational Pathology: A Clinical Data Perspective](https://mlsanity.com/paper/2407.07841/) (2024)
- [Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions](https://mlsanity.com/paper/2408.09476/) (2024)
- [Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: Achieving SOTA predictive performance with fewer data using Swin Transformer](https://mlsanity.com/paper/2208.10495/) (2022)
- [Multi-Scale Relational Graph Convolutional Network for Multiple Instance Learning in Histopathology Images](https://mlsanity.com/paper/2212.08781/) (2022)
- [Fully transformer-based biomarker prediction from colorectal cancer histology: a large-scale multicentric study](https://mlsanity.com/paper/2301.09617/) (2023)
- [Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings](https://mlsanity.com/paper/2511.23204/) (2025)
- [GRASP: GRAph-Structured Pyramidal Whole Slide Image Representation](https://mlsanity.com/paper/2402.03592/) (2024)
- [Ensemble learning of pathology foundation models for precision oncology](https://mlsanity.com/paper/2508.16085/) (2025)
- [CIMIL-CRC: a clinically-informed multiple instance learning framework for patient-level colorectal cancer molecular subtypes classification from H&E stained images](https://mlsanity.com/paper/2401.16131/) (2024)
- [Federated attention consistent learning models for prostate cancer diagnosis and Gleason grading](https://mlsanity.com/paper/2302.06089/) (2023)
- [Towards Cellular-Scale Interpretability in Pathology Foundation Models for Biomarker Assessment](https://mlsanity.com/paper/2511.05150/) (2025)
- [Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology](https://mlsanity.com/paper/2603.08328/) (2026)
- [Development and Validation of a Deep Learning-Based Microsatellite Instability Predictor from Prostate Cancer Whole-Slide Images](https://mlsanity.com/paper/2310.08743/) (2023)
- [Multi-Cohort Framework with Cohort-Aware Attention and Adversarial Mutual-Information Minimization for Whole Slide Image Classification](https://mlsanity.com/paper/2409.11119/) (2024)
- [Exploring the Interplay Between Colorectal Cancer Subtypes Genomic Variants and Cellular Morphology: A Deep-Learning Approach](https://mlsanity.com/paper/2303.14703/) (2023)
- [Multi-Beholder: Biomarker Prediction for Low-Grade Glioma with Multiple Instance Learning and One-Class Classification](https://mlsanity.com/paper/2310.07464/) (2023)
- [Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis](https://mlsanity.com/paper/2506.05184/) (2025)
- [A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI](https://mlsanity.com/paper/2512.21372/) (2025)
- [Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency](https://mlsanity.com/paper/2602.00151/) (2026)
- [Conserved Immune Topology Improves Pathology Foundation Model Generalization for Cross-Cancer MSI-H Prediction](https://mlsanity.com/paper/2609.05182/) (2026)

## Abstract (arXiv)

We propose a Deep learning-based weak label learning method for analyzing whole slide images (WSIs) of Hematoxylin and Eosin (H&E) stained tumor tissue not requiring pixel-level or tile-level annotations using Self-supervised pre-training and heterogeneity-aware deep Multiple Instance LEarning (DeepSMILE). We apply DeepSMILE to the task of Homologous recombination deficiency (HRD) and microsatellite instability (MSI) prediction. We utilize contrastive self-supervised learning to pre-train a feature extractor on histopathology tiles of cancer tissue. Additionally, we use variability-aware deep multiple instance learning to learn the tile feature aggregation function while modeling tumor heterogeneity. For MSI prediction in a tumor-annotated and color normalized subset of TCGA-CRC (n=360 patients), contrastive self-supervised learning improves the tile supervision baseline from 0.77 to 0.87 AUROC, on par with our proposed DeepSMILE method. On TCGA-BC (n=1041 patients) without any manual annotations, DeepSMILE improves HRD classification performance from 0.77 to 0.81 AUROC compared to tile supervision with either a self-supervised or ImageNet pre-trained feature extractor. Our proposed methods reach the baseline performance using only 40% of the labeled data on both datasets. These improvements suggest we can use standard self-supervised learning techniques combined with multiple instance learning in the histopathology domain to improve genomic label classification performance with fewer labeled data.