# Handcrafted Histological Transformer (H2T): Unsupervised Representation of Whole Slide Images

arXiv: https://arxiv.org/abs/2202.07001 · HTML: https://mlsanity.com/paper/2202.07001/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Multi-task Self-Supervised Visual Learning](https://mlsanity.com/paper/1708.07860/) (2017)
- [SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning](https://mlsanity.com/paper/1911.04623/) (2019)
- [Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification](https://mlsanity.com/paper/1504.07947/) (2015)
- [Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles](https://mlsanity.com/paper/1811.09795/) (2018)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [On Robustness and Transferability of Convolutional Neural Networks](https://mlsanity.com/paper/2007.08558/) (2020)
- [Training Vision Transformers for Image Retrieval](https://mlsanity.com/paper/2102.05644/) (2021)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images](https://mlsanity.com/paper/2001.01599/) (2020)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [Dense Steerable Filter CNNs for Exploiting Rotational Symmetry in Histology Images](https://mlsanity.com/paper/2004.03037/) (2020)
- [Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imaging](https://mlsanity.com/paper/2111.01556/) (2021)
- [Divide-and-Rule: Self-Supervised Learning for Survival Analysis in Colorectal Cancer](https://mlsanity.com/paper/2007.03292/) (2020)

## Built on by (most cited first, 30 of 32)

- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11643/) (2024)
- [Multimodal Prototyping for cancer survival prediction](https://mlsanity.com/paper/2407.00224/) (2024)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [CoNIC Challenge: Pushing the Frontiers of Nuclear Detection, Segmentation, Classification and Counting](https://mlsanity.com/paper/2303.06274/) (2023)
- [Domain Generalization in Computational Pathology: Survey and Guidelines](https://mlsanity.com/paper/2310.19656/) (2023)
- [Multistain Pretraining for Slide Representation Learning in Pathology](https://mlsanity.com/paper/2408.02859/) (2024)
- [CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment](https://mlsanity.com/paper/2406.05205/) (2024)
- [Masked Pre-Training of Transformers for Histology Image Analysis](https://mlsanity.com/paper/2304.07434/) (2023)
- [StainFuser: Controlling Diffusion for Faster Neural Style Transfer in Multi-Gigapixel Histology Images](https://mlsanity.com/paper/2403.09302/) (2024)
- [Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation](https://mlsanity.com/paper/2504.18856/) (2025)
- [DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction](https://mlsanity.com/paper/2510.00053/) (2025)
- [Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learning](https://mlsanity.com/paper/2603.22198/) (2026)
- [Unsupervised Mutual Transformer Learning for Multi-Gigapixel Whole Slide Image Classification](https://mlsanity.com/paper/2305.02032/) (2023)
- [Pan-cancer Histopathology WSI Pre-training with Position-aware Masked Autoencoder](https://mlsanity.com/paper/2407.07504/) (2024)
- [PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization](https://mlsanity.com/paper/2503.06482/) (2025)
- [PTCMIL: Multiple Instance Learning via Prompt Token Clustering for Whole Slide Image Analysis](https://mlsanity.com/paper/2507.18848/) (2025)
- [PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction](https://mlsanity.com/paper/2509.20022/) (2025)
- [CD-Net: Histopathology Representation Learning using Pyramidal Context-Detail Network](https://mlsanity.com/paper/2203.15078/) (2022)
- [HistoPerm: A Permutation-Based View Generation Approach for Improving Histopathologic Feature Representation Learning](https://mlsanity.com/paper/2209.06185/) (2022)
- [Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning](https://mlsanity.com/paper/2407.14904/) (2024)
- [Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining](https://mlsanity.com/paper/2605.16392/) (2026)
- [Prototype-driven fusion of pathology and spatial transcriptomics for interpretable survival prediction](https://mlsanity.com/paper/2602.12441/) (2026)
- [Classifying Whole Slide Images: What Matters?](https://mlsanity.com/paper/2310.03279/) (2023)
- [Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images](https://mlsanity.com/paper/2503.10731/) (2025)
- [Histomorphology-Guided Prototypical Multi-Instance Learning for Breast Cancer WSI Classification](https://mlsanity.com/paper/2503.17983/) (2025)
- [Cross-Modal Prototype Allocation: Unsupervised Slide Representation Learning via Patch-Text Contrast in Computational Pathology](https://mlsanity.com/paper/2503.20190/) (2025)
- [The Next Layer: Augmenting Foundation Models with Structure-Preserving and Attention-Guided Learning for Local Patches to Global Context Awareness in Computational Pathology](https://mlsanity.com/paper/2508.19914/) (2025)
- [Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis](https://mlsanity.com/paper/2512.01116/) (2025)
- [Tissue Classification and Whole-Slide Images Analysis via Modeling of the Tumor Microenvironment and Biological Pathways](https://mlsanity.com/paper/2601.08336/) (2026)

## Abstract (arXiv)

Diagnostic, prognostic and therapeutic decision-making of cancer in pathology clinics can now be carried out based on analysis of multi-gigapixel tissue images, also known as whole-slide images (WSIs). Recently, deep convolutional neural networks (CNNs) have been proposed to derive unsupervised WSI representations; these are attractive as they rely less on expert annotation which is cumbersome. However, a major trade-off is that higher predictive power generally comes at the cost of interpretability, posing a challenge to their clinical use where transparency in decision-making is generally expected. To address this challenge, we present a handcrafted framework based on deep CNN for constructing holistic WSI-level representations. Building on recent findings about the internal working of the Transformer in the domain of natural language processing, we break down its processes and handcraft them into a more transparent framework that we term as the Handcrafted Histological Transformer or H2T. Based on our experiments involving various datasets consisting of a total of 5,306 WSIs, the results demonstrate that H2T based holistic WSI-level representations offer competitive performance compared to recent state-of-the-art methods and can be readily utilized for various downstream analysis tasks. Finally, our results demonstrate that the H2T framework can be up to 14 times faster than the Transformer models.