# ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification

arXiv: https://arxiv.org/abs/2202.07570 · HTML: https://mlsanity.com/paper/2202.07570/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification](https://mlsanity.com/paper/1504.07947/) (2015)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias](https://mlsanity.com/paper/2001.03152/) (2020)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Learning with Differentiable Perturbed Optimizers](https://mlsanity.com/paper/2002.08676/) (2020)
- [TransMix: Attend to Mix for Vision Transformers](https://mlsanity.com/paper/2111.09833/) (2021)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)
- [Group Equivariant Stand-Alone Self-Attention For Vision](https://mlsanity.com/paper/2010.00977/) (2020)
- [Hierarchical Graph Representations in Digital Pathology](https://mlsanity.com/paper/2102.11057/) (2021)

## Built on by (most cited first, 21 of 21)

- [Multiple Instance Learning for Digital Pathology: A Review on the State-of-the-Art, Limitations & Future Potential](https://mlsanity.com/paper/2206.04425/) (2022)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders](https://mlsanity.com/paper/2310.20704/) (2023)
- [Equipping Computational Pathology Systems with Artifact Processing Pipelines: A Showcase for Computation and Performance Trade-offs](https://mlsanity.com/paper/2403.07743/) (2024)
- [LatentDR: Improving Model Generalization Through Sample-Aware Latent Degradation and Restoration](https://mlsanity.com/paper/2308.14596/) (2023)
- [Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning](https://mlsanity.com/paper/2309.06439/) (2023)
- [CellMix: A General Instance Relationship based Method for Data Augmentation Towards Pathology Image Classification](https://mlsanity.com/paper/2301.11513/) (2023)
- [MMLNB: Multi-Modal Learning for Neuroblastoma Subtyping Classification Assisted with Textual Description Generation](https://mlsanity.com/paper/2503.12927/) (2025)
- [CD-Net: Histopathology Representation Learning using Pyramidal Context-Detail Network](https://mlsanity.com/paper/2203.15078/) (2022)
- [Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review](https://mlsanity.com/paper/2211.14847/) (2022)
- [Self-supervised learning-based cervical cytology for the triage of HPV-positive women in resource-limited settings and low-data regime](https://mlsanity.com/paper/2302.05195/) (2023)
- [SF2Former: Amyotrophic Lateral Sclerosis Identification From Multi-center MRI Data Using Spatial and Frequency Fusion Transformer](https://mlsanity.com/paper/2302.10859/) (2023)
- [Beyond Labels: A Self-Supervised Framework with Masked Autoencoders and Random Cropping for Breast Cancer Subtype Classification](https://mlsanity.com/paper/2410.12006/) (2024)
- [Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Review](https://mlsanity.com/paper/2410.19820/) (2024)
- [Multi-Scale Hybrid Vision Transformer for Learning Gastric Histology: AI-Based Decision Support System for Gastric Cancer Treatment](https://mlsanity.com/paper/2202.08510/) (2022)
- [Improved Breast Cancer Diagnosis through Transfer Learning on Hematoxylin and Eosin Stained Histology Images](https://mlsanity.com/paper/2309.08745/) (2023)
- [Classifying Whole Slide Images: What Matters?](https://mlsanity.com/paper/2310.03279/) (2023)
- [ITC-RWKV: Interactive Tissue-Cell Modeling with Recurrent Key-Value Aggregation for Histopathological Subtyping](https://mlsanity.com/paper/2510.21479/) (2025)
- [SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classification](https://mlsanity.com/paper/2604.15711/) (2026)
- [Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology](https://mlsanity.com/paper/2609.18578/) (2026)

## Abstract (arXiv)

Progress in digital pathology is hindered by high-resolution images and the prohibitive cost of exhaustive localized annotations. The commonly used paradigm to categorize pathology images is patch-based processing, which often incorporates multiple instance learning (MIL) to aggregate local patch-level representations yielding image-level prediction. Nonetheless, diagnostically relevant regions may only take a small fraction of the whole tissue, and current MIL-based approaches often process images uniformly, discarding the inter-patches interactions. To alleviate these issues, we propose ScoreNet, a new efficient transformer that exploits a differentiable recommendation stage to extract discriminative image regions and dedicate computational resources accordingly. The proposed transformer leverages the local and global attention of a few dynamically recommended high-resolution regions at an efficient computational cost. We further introduce a novel mixing data-augmentation, namely ScoreMix, by leveraging the image's semantic distribution to guide the data mixing and produce coherent sample-label pairs. ScoreMix is embarrassingly simple and mitigates the pitfalls of previous augmentations, which assume a uniform semantic distribution and risk mislabeling the samples. Thorough experiments and ablation studies on three breast cancer histology datasets of Haematoxylin & Eosin (H&E) have validated the superiority of our approach over prior arts, including transformer-based models on tumour regions-of-interest (TRoIs) classification. ScoreNet equipped with proposed ScoreMix augmentation demonstrates better generalization capabilities and achieves new state-of-the-art (SOTA) results with only 50% of the data compared to other mixing augmentation variants. Finally, ScoreNet yields high efficacy and outperforms SOTA efficient transformers, namely TransPath and SwinTransformer.