# A graph-transformer for whole slide image classification

arXiv: https://arxiv.org/abs/2205.09671 · HTML: https://mlsanity.com/paper/2205.09671/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers](https://mlsanity.com/paper/1604.00825/) (2016)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification](https://mlsanity.com/paper/1504.07947/) (2015)
- [Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks](https://mlsanity.com/paper/2107.13048/) (2021)
- [CGC-Net: Cell Graph Convolutional Network for Grading of Colorectal Cancer Histology Images](https://mlsanity.com/paper/1909.01068/) (2019)
- [Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks](https://mlsanity.com/paper/1901.11489/) (2019)
- [Representation Learning of Histopathology Images using Graph Neural Networks](https://mlsanity.com/paper/2004.07399/) (2020)
- [Computational Pathology: Challenges and Promises for Tissue Analysis](https://mlsanity.com/paper/1601.00027/) (2015)

## Built on by (most cited first, 30 of 84)

- [Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2403.07719/) (2024)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [Histopathology Whole Slide Image Analysis with Heterogeneous Graph Representation Learning](https://mlsanity.com/paper/2307.04189/) (2023)
- [PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology](https://mlsanity.com/paper/2502.08916/) (2025)
- [ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2502.08391/) (2025)
- [Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2301.08125/) (2023)
- [SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology](https://mlsanity.com/paper/2312.15010/) (2023)
- [Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Good Instance Classifier is All You Need](https://mlsanity.com/paper/2307.02249/) (2023)
- [An Aggregation of Aggregation Methods in Computational Pathology](https://mlsanity.com/paper/2211.01256/) (2022)
- [FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2411.14743/) (2024)
- [HIGT: Hierarchical Interaction Graph-Transformer for Whole Slide Image Analysis](https://mlsanity.com/paper/2309.07400/) (2023)
- [Large-scale Long-tailed Disease Diagnosis on Radiology Images](https://mlsanity.com/paper/2312.16151/) (2023)
- [MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning](https://mlsanity.com/paper/2408.11505/) (2024)
- [Structure Embedded Nucleus Classification for Histopathology Images](https://mlsanity.com/paper/2302.11416/) (2023)
- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images](https://mlsanity.com/paper/2305.05314/) (2023)
- [Weakly Supervised Joint Whole-Slide Segmentation and Classification in Prostate Cancer](https://mlsanity.com/paper/2301.02933/) (2023)
- [FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer](https://mlsanity.com/paper/2403.12821/) (2024)
- [Benchmarking Embedding Aggregation Methods in Computational Pathology: A Clinical Data Perspective](https://mlsanity.com/paper/2407.07841/) (2024)
- [Sm: enhanced localization in Multiple Instance Learning for medical imaging classification](https://mlsanity.com/paper/2410.03276/) (2024)
- [Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy](https://mlsanity.com/paper/2306.07999/) (2023)
- [Multi-Scale Prototypical Transformer for Whole Slide Image Classification](https://mlsanity.com/paper/2307.02308/) (2023)
- [Cell Graph Transformer for Nuclei Classification](https://mlsanity.com/paper/2402.12946/) (2024)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [Combining Graph Neural Network and Mamba to Capture Local and Global Tissue Spatial Relationships in Whole Slide Images](https://mlsanity.com/paper/2406.04377/) (2024)
- [GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation](https://mlsanity.com/paper/2407.05540/) (2024)
- [TractGraphFormer: Anatomically Informed Hybrid Graph CNN-Transformer Network for Interpretable Sex and Age Prediction from Diffusion MRI Tractography](https://mlsanity.com/paper/2407.08883/) (2024)
- [Multi-Scale Relational Graph Convolutional Network for Multiple Instance Learning in Histopathology Images](https://mlsanity.com/paper/2212.08781/) (2022)
- [Revisiting Computer-Aided Tuberculosis Diagnosis](https://mlsanity.com/paper/2307.02848/) (2023)
- [Integrative Graph-Transformer Framework for Histopathology Whole Slide Image Representation and Classification](https://mlsanity.com/paper/2403.18134/) (2024)

## Abstract (arXiv)

Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected 4,818 WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy= 91.2 ± 2.5%) based on five-fold cross-validation, and mean accuracy = 82.3 ± 1.0% on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.