mlsanityData to 7 Oct 2026

Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning

arXiv 2206.02647 ↗First posted 6 Jun 2022cs.CVcited by 357

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) and their multi-scale and hierarchical variations have been successful at capturing image representations but their use has been generally studied for low-resolution images (e.g. - 256x256, 384384). For gigapixel whole-slide imaging (WSI) in computational pathology, WSIs can be as large as 150000x150000 pixels at 20X magnification and exhibit a hierarchical structure of visual tokens across varying resolutions: from 16x16 images capture spatial patterns among cells, to 4096x4096 images characterizing interactions within the tissue microenvironment. We introduce a new ViT architecture called the Hierarchical Image Pyramid Transformer (HIPT), which leverages the natural hierarchical structure inherent in WSIs using two levels of self-supervised learning to learn high-resolution image representations. HIPT is pretrained across 33 cancer types using 10,678 gigapixel WSIs, 408,218 4096x4096 images, and 104M 256x256 images. We benchmark HIPT representations on 9 slide-level tasks, and demonstrate that: 1) HIPT with hierarchical pretraining outperforms current state-of-the-art methods for cancer subtyping and survival prediction, 2) self-supervised ViTs are able to model important inductive biases about the hierarchical structure of phenotypes in the tumor microenvironment.

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). Builds on 46 of its 88 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

46 of 88 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2017Deep Setscited by 1,394
  10. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  11. 2021Multiscale Vision Transformerscited by 896
  12. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
34 more
  1. 2021Transformer in Transformercited by 632
  2. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  3. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  4. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  5. 2018HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Imagescited by 387
  6. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  7. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  8. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  9. 2020Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networkscited by 178
  10. 2015Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classificationcited by 159
  11. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  12. 2020Self supervised contrastive learning for digital histopathologycited by 146
  13. 2021BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Imagescited by 130
  14. 2021How to represent part-whole hierarchies in a neural networkcited by 123
  15. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  16. 2016Towards a Neural Statisticiancited by 119
  17. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  18. 2020Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primarycited by 89
  19. 2020Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotationscited by 89
  20. 2020Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Imagescited by 81
  21. 2019CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentationcited by 73
  22. 2018Neural Image Compression for Gigapixel Histopathology Image Analysiscited by 65
  23. 2021Hierarchical Graph Representations in Digital Pathologycited by 58
  24. 2022Self-Supervised Vision Transformers Learn Visual Concepts in Histopathologycited by 49
  25. 2019Deep learning-based survival prediction for multiple cancer types using histopathology imagescited by 47
  26. 2020Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histologycited by 43
  27. 2020HACT-Net: A Hierarchical Cell-to-Tissue Graph Neural Network for Histopathological Image Classificationcited by 33
  28. 2020Interpretable Survival Prediction for Colorectal Cancer using Deep Learningcited by 30
  29. 2021Self-Supervised Representation Learning using Visual Field Expansion on Digital Pathologycited by 25
  30. 2021Multi-Class Cell Detection Using Spatial Context Representationcited by 24
  31. 2021Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancerscited by 23
  32. 2022HiP: Hierarchical Perceivercited by 15
  33. 2019Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor Infiltrating Lymphocytes in Invasive Breast Cancercited by 12
  34. 2017Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Imagescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 90; 2024: 102; 2025: 91; 2026: 57.

357 papers.

Sort:

By area:

  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  3. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  4. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  5. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  6. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  7. 2023CellViT: Vision Transformers for Precise Cell Segmentation and Classificationcited by 108
  8. 2023Artificial Intelligence for Digital and Computational Pathologycited by 106
  9. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 86
  10. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
  11. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  12. 2024Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathologycited by 70
38 more
  1. 2024Phikon-v2, A large and public feature extractor for biomarker predictioncited by 67
  2. 2024Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathologycited by 66
  3. 2024Dynamic Graph Representation with Knowledge-aware Attention for Histopathology Whole Slide Image Analysiscited by 66
  4. 2023Attention-Challenging Multiple Instance Learning for Whole Slide Image Classificationcited by 61
  5. 2023Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classificationcited by 58
  6. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  7. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  8. 2024Accurate Spatial Gene Expression Prediction by integrating Multi-resolution featurescited by 52
  9. 2024Hibou: A Family of Foundational Vision Transformers for Pathologycited by 52
  10. 2024Transcriptomics-guided Slide Representation Learning in Computational Pathologycited by 50
  11. 2023Histopathology Whole Slide Image Analysis with Heterogeneous Graph Representation Learningcited by 47
  12. 2023Domain Generalization for Medical Image Analysis: A Reviewcited by 47
  13. 2022Deep Learning-enabled Virtual Histological Staining of Biological Samplescited by 42
  14. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  15. 2023The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classificationcited by 41
  16. 2023Structured State Space Models for Multiple Instance Learning in Digital Pathologycited by 41
  17. 2024Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Predictioncited by 41
  18. 2024RudolfV: A Foundation Model by Pathologists for Pathologistscited by 40
  19. 2023Learned representation-guided diffusion models for large-image generationcited by 39
  20. 2025PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathologycited by 36
  21. 20242DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classificationcited by 35
  22. 2024WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answeringcited by 34
  23. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  24. 2023MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosiscited by 33
  25. 2023WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Imagescited by 31
  26. 2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
  27. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 29
  28. 2023DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathologycited by 28
  29. 2025Do Multiple Instance Learning Models Transfer?cited by 28
  30. 2023Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classificationcited by 27
  31. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  32. 2023SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathologycited by 26
  33. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  34. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
  35. 2022Giga-SSL: Self-Supervised Learning for Gigapixel Imagescited by 24
  36. 2022AdvMIL: Adversarial Multiple Instance Learning for the Survival Analysis on Whole-Slide Imagescited by 24
  37. 2023Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Good Instance Classifier is All You Needcited by 21
  38. 2024Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosiscited by 21

50 most cited of 357 All 357 (JSON)