# HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival Prediction from Whole Slide Image

arXiv: https://arxiv.org/abs/2306.17373 · HTML: https://mlsanity.com/paper/2306.17373/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks](https://mlsanity.com/paper/2107.13048/) (2021)
- [Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer](https://mlsanity.com/paper/2106.03650/) (2021)
- [Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images](https://mlsanity.com/paper/2001.01599/) (2020)
- [CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentation](https://mlsanity.com/paper/1908.10555/) (2019)
- [Weakly supervised multiple instance learning histopathological tumor segmentation](https://mlsanity.com/paper/2004.05024/) (2020)
- [Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides](https://mlsanity.com/paper/1811.08513/) (2018)
- [Divide-and-Rule: Self-Supervised Learning for Survival Analysis in Colorectal Cancer](https://mlsanity.com/paper/2007.03292/) (2020)
- [Incorporating intratumoral heterogeneity into weakly-supervised deep learning models via variance pooling](https://mlsanity.com/paper/2206.08885/) (2022)
- [Multiple Instance Learning with Mixed Supervision in Gleason Grading](https://mlsanity.com/paper/2206.12798/) (2022)

## Built on by (most cited first, 14 of 14)

- [SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding](https://mlsanity.com/paper/2410.11761/) (2024)
- [MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning](https://mlsanity.com/paper/2408.11505/) (2024)
- [MiCo: Multiple Instance Learning with Context-Aware Clustering for Whole Slide Image Analysis](https://mlsanity.com/paper/2506.18028/) (2025)
- [PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction](https://mlsanity.com/paper/2509.20022/) (2025)
- [SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification](https://mlsanity.com/paper/2407.08968/) (2024)
- [Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2410.10260/) (2024)
- [Efficient Multi-Slide Visual-Language Feature Fusion for Placental Disease Classification](https://mlsanity.com/paper/2508.03277/) (2025)
- [Point Transformer with Federated Learning for Predicting Breast Cancer HER2 Status from Hematoxylin and Eosin-Stained Whole Slide Images](https://mlsanity.com/paper/2312.06454/) (2023)
- [Cross-Cancer Knowledge Transfer in WSI-based Prognosis Prediction](https://mlsanity.com/paper/2508.13482/) (2025)
- [Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis](https://mlsanity.com/paper/2511.18089/) (2025)
- [Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis](https://mlsanity.com/paper/2512.01116/) (2025)
- [Efficient AI-Driven Multi-Section Whole Slide Image Analysis for Biochemical Recurrence Prediction in Prostate Cancer](https://mlsanity.com/paper/2603.20273/) (2026)
- [Graph Mamba Survival Analysis Based on Topology-Aware ordering](https://mlsanity.com/paper/2606.02602/) (2026)
- [Morphology signal in whole slide image foundation models can automatically triage slides](https://mlsanity.com/paper/2609.01987/) (2026)

## Abstract (arXiv)

Survival prediction based on whole slide images (WSIs) is a challenging task for patient-level multiple instance learning (MIL). Due to the vast amount of data for a patient (one or multiple gigapixels WSIs) and the irregularly shaped property of WSI, it is difficult to fully explore spatial, contextual, and hierarchical interaction in the patient-level bag. Many studies adopt random sampling pre-processing strategy and WSI-level aggregation models, which inevitably lose critical prognostic information in the patient-level bag. In this work, we propose a hierarchical vision Transformer framework named HVTSurv, which can encode the local-level relative spatial information, strengthen WSI-level context-aware communication, and establish patient-level hierarchical interaction. Firstly, we design a feature pre-processing strategy, including feature rearrangement and random window masking. Then, we devise three layers to progressively obtain patient-level representation, including a local-level interaction layer adopting Manhattan distance, a WSI-level interaction layer employing spatial shuffle, and a patient-level interaction layer using attention pooling. Moreover, the design of hierarchical network helps the model become more computationally efficient. Finally, we validate HVTSurv with 3,104 patients and 3,752 WSIs across 6 cancer types from The Cancer Genome Atlas (TCGA). The average C-Index is 2.50-11.30% higher than all the prior weakly supervised methods over 6 TCGA datasets. Ablation study and attention visualization further verify the superiority of the proposed HVTSurv. Implementation is available at: https://github.com/szc19990412/HVTSurv.