# Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology

arXiv: https://arxiv.org/abs/2203.00585 · HTML: https://mlsanity.com/paper/2203.00585/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks](https://mlsanity.com/paper/2107.13048/) (2021)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology](https://mlsanity.com/paper/2012.03583/) (2020)
- [Self-supervised driven consistency training for annotation efficient histopathology image analysis](https://mlsanity.com/paper/2102.03897/) (2021)
- [Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding](https://mlsanity.com/paper/1910.10825/) (2019)
- [Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers](https://mlsanity.com/paper/2109.05819/) (2021)
- [Weakly Supervised Prostate TMA Classification via Graph Convolutional Networks](https://mlsanity.com/paper/1910.13328/) (2019)
- [Adversarial learning of cancer tissue representations](https://mlsanity.com/paper/2108.02223/) (2021)
- [Resource and data efficient self supervised learning](https://mlsanity.com/paper/2109.01721/) (2021)

## Built on by (most cited first, 30 of 50)

- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [Phikon-v2, A large and public feature extractor for biomarker prediction](https://mlsanity.com/paper/2409.09173/) (2024)
- [Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification](https://mlsanity.com/paper/2303.08446/) (2023)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification](https://mlsanity.com/paper/2305.17891/) (2023)
- [RudolfV: A Foundation Model by Pathologists for Pathologists](https://mlsanity.com/paper/2401.04079/) (2024)
- [Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis](https://mlsanity.com/paper/2208.08789/) (2022)
- [Multiple Instance Learning via Iterative Self-Paced Supervised Contrastive Learning](https://mlsanity.com/paper/2210.09452/) (2022)
- [Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Good Instance Classifier is All You Need](https://mlsanity.com/paper/2307.02249/) (2023)
- [Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders](https://mlsanity.com/paper/2310.20704/) (2023)
- [Domain-specific optimization and diverse evaluation of self-supervised models for histopathology](https://mlsanity.com/paper/2310.13259/) (2023)
- [From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathology](https://mlsanity.com/paper/2204.05044/) (2022)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [AI-based Anomaly Detection for Clinical-Grade Histopathological Diagnostics](https://mlsanity.com/paper/2406.14866/) (2024)
- [HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization](https://mlsanity.com/paper/2407.03307/) (2024)
- [Self-Supervision Enhances Instance-based Multiple Instance Learning Methods in Digital Pathology: A Benchmark Study](https://mlsanity.com/paper/2505.01109/) (2025)
- [OpenAL: An Efficient Deep Active Learning Framework for Open-Set Pathology Image Classification](https://mlsanity.com/paper/2307.05254/) (2023)
- [Lifelong Histopathology Whole Slide Image Retrieval via Distance Consistency Rehearsal](https://mlsanity.com/paper/2407.08153/) (2024)
- [Towards Effective and Efficient Context-aware Nucleus Detection in Histopathology Whole Slide Images](https://mlsanity.com/paper/2503.05678/) (2025)
- [Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning](https://mlsanity.com/paper/2309.06439/) (2023)
- [Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning](https://mlsanity.com/paper/2403.14616/) (2024)
- [Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detection](https://mlsanity.com/paper/2504.13186/) (2025)
- [Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis](https://mlsanity.com/paper/2207.11581/) (2022)
- [Multi-modal Masked Autoencoders Learn Compositional Histopathological Representations](https://mlsanity.com/paper/2209.01534/) (2022)
- [Out-Of-Domain Unlabeled Data Improves Generalization](https://mlsanity.com/paper/2310.00027/) (2023)
- [BioFusionNet: Deep Learning-Based Survival Risk Stratification in ER+ Breast Cancer Through Multifeature and Multimodal Data Fusion](https://mlsanity.com/paper/2402.10717/) (2024)
- [Patient-level Microsatellite Stability Assessment from Whole Slide Images By Combining Momentum Contrast Learning and Group Patch Embeddings](https://mlsanity.com/paper/2208.10429/) (2022)
- [HistoPerm: A Permutation-Based View Generation Approach for Improving Histopathologic Feature Representation Learning](https://mlsanity.com/paper/2209.06185/) (2022)
- [Genomics-guided Representation Learning for Pathologic Pan-cancer Tumor Microenvironment Subtype Prediction](https://mlsanity.com/paper/2406.06517/) (2024)

## Abstract (arXiv)

Tissue phenotyping is a fundamental task in learning objective characterizations of histopathologic biomarkers within the tumor-immune microenvironment in cancer pathology. However, whole-slide imaging (WSI) is a complex computer vision in which: 1) WSIs have enormous image resolutions with precludes large-scale pixel-level efforts in data curation, and 2) diversity of morphological phenotypes results in inter- and intra-observer variability in tissue labeling. To address these limitations, current efforts have proposed using pretrained image encoders (transfer learning from ImageNet, self-supervised pretraining) in extracting morphological features from pathology, but have not been extensively validated. In this work, we conduct a search for good representations in pathology by training a variety of self-supervised models with validation on a variety of weakly-supervised and patch-level tasks. Our key finding is in discovering that Vision Transformers using DINO-based knowledge distillation are able to learn data-efficient and interpretable features in histology images wherein the different attention heads learn distinct morphological phenotypes. We make evaluation code and pretrained weights publicly-available at: https://github.com/Richarizardd/Self-Supervised-ViT-Path.