# Feature Learning with Multi-Stage Vision Transformers on Inter-Modality HER2 Status Scoring and Tumor Classification on Whole Slides

arXiv: https://arxiv.org/abs/2512.22335 · HTML: https://mlsanity.com/paper/2512.22335/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 50 references that are in the map.
- First posted on arXiv 26 Dec 2025.

## Builds on (most cited first, 11 of 11)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Vision Transformers with Hierarchical Attention](https://mlsanity.com/paper/2106.03180/) (2021)
- [Predicting Ki67, ER, PR, and HER2 Statuses from H&E-stained Breast Cancer Images](https://mlsanity.com/paper/2308.01982/) (2023)
- [Vision Transformers for Small Histological Datasets Learned through Knowledge Distillation](https://mlsanity.com/paper/2305.17370/) (2023)
- [HER2 and FISH Status Prediction in Breast Biopsy H&E-Stained Images Using Deep Learning](https://mlsanity.com/paper/2408.13818/) (2024)
- [DIOR-ViT: Differential Ordinal Learning Vision Transformer for Cancer Classification in Pathology Images](https://mlsanity.com/paper/2407.08503/) (2024)
- [Hierarchical Vision Transformers for Context-Aware Prostate Cancer Grading in Whole Slide Images](https://mlsanity.com/paper/2312.12619/) (2023)
- [Automated HER2 Scoring in Breast Cancer Images Using Deep Learning and Pyramid Sampling](https://mlsanity.com/paper/2404.00837/) (2024)
- [Interpretable HER2 scoring by evaluating clinical Guidelines through a weakly supervised, constrained Deep Learning Approach](https://mlsanity.com/paper/2211.09559/) (2022)

## Abstract (arXiv)

The popular use of histopathology images, such as hematoxylin and eosin (H&E), has proven to be useful in detecting tumors. However, moving such cancer cases forward for treatment requires accurate on the amount of the human epidermal growth factor receptor 2 (HER2) protein expression. Predicting both the lower and higher levels of HER2 can be challenging. Moreover, jointly analyzing H&E and immunohistochemistry (IHC) stained images for HER2 scoring is difficult. Although several deep learning methods have been investigated to address the challenge of HER2 scoring, they suffer from providing a pixel-level localization of HER2 status. In this study, we propose a single end-to-end pipeline using a system of vision transformers with HER2 status scoring on whole slide images of WSIs. The method includes patch-wise processing of H&E WSIs for tumor localization. A novel mapping function is proposed to correspondingly identify correlated IHC WSIs regions with malignant regions on H&E. A clinically inspired HER2 scoring mechanism is embedded in the pipeline and allows for automatic pixel-level annotation of 4-way HER2 scoring (0, 1+, 2+, and 3+). Also, the proposed method accurately returns HER2-negative and HER2-positive. Privately curated datasets were collaboratively extracted from 13 different cases of WSIs of H&E and IHC. A thorough experiment is conducted on the proposed method. Results obtained showed a good classification accuracy during tumor localization. Also, a classification accuracy of 0.94 and a specificity of 0.933 were returned for the prediction of HER2 status, scoring in the 4-way methods. The applicability of the proposed pipeline was investigated using WSIs patches as comparable to human pathologists. Findings from the study showed the usability of jointly evaluated H&E and IHC images on end-to-end ViTs-based models for HER2 scoring