# Vision Transformers for Small Histological Datasets Learned through Knowledge Distillation

arXiv: https://arxiv.org/abs/2305.17370 · HTML: https://mlsanity.com/paper/2305.17370/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 27 May 2023.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Multiresolution Knowledge Distillation for Anomaly Detection](https://mlsanity.com/paper/2011.11108/) (2020)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Attention-based Clinical Note Summarization](https://mlsanity.com/paper/2104.08942/) (2021)
- [MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data](https://mlsanity.com/paper/2104.03618/) (2021)
- [Detection and Localization of Melanoma Skin Cancer in Histopathological Whole Slide Images](https://mlsanity.com/paper/2302.03014/) (2023)
- [Multi-input segmentation of damaged brain in acute ischemic stroke patients using slow fusion with skip connection](https://mlsanity.com/paper/2203.10039/) (2022)
- [Classification of Breast Cancer Histology using Deep Learning](https://mlsanity.com/paper/1802.08080/) (2018)

## Built on by (most cited first, 10 of 10)

- [Equipping Computational Pathology Systems with Artifact Processing Pipelines: A Showcase for Computation and Performance Trade-offs](https://mlsanity.com/paper/2403.07743/) (2024)
- [Bridging Simplicity and Sophistication using GLinear: A Novel Architecture for Enhanced Time Series Prediction](https://mlsanity.com/paper/2501.01087/) (2025)
- [Siamese Content-based Search Engine for a More Transparent Skin and Breast Cancer Diagnosis through Histological Imaging](https://mlsanity.com/paper/2401.08272/) (2024)
- [Balancing Privacy and Progress in Artificial Intelligence: Anonymization in Histopathology for Biomedical Research and Education](https://mlsanity.com/paper/2307.09426/) (2023)
- [Enhancing Prostate Cancer Diagnosis with Deep Learning: A Study using mpMRI Segmentation and Classification](https://mlsanity.com/paper/2310.05371/) (2023)
- [Learning using privileged information for segmenting tumors on digital mammograms](https://mlsanity.com/paper/2402.06379/) (2024)
- [EFCM: Efficient Fine-tuning on Compressed Models for deployment of large models in medical image analysis](https://mlsanity.com/paper/2409.11817/) (2024)
- [Dataset Distillation with Probabilistic Latent Features](https://mlsanity.com/paper/2505.06647/) (2025)
- [Automated Quality Evaluation of Cervical Cytopathology Whole Slide Images Based on Content Analysis](https://mlsanity.com/paper/2505.13875/) (2025)
- [Feature Learning with Multi-Stage Vision Transformers on Inter-Modality HER2 Status Scoring and Tumor Classification on Whole Slides](https://mlsanity.com/paper/2512.22335/) (2025)

## Abstract (arXiv)

Computational Pathology (CPATH) systems have the potential to automate diagnostic tasks. However, the artifacts on the digitized histological glass slides, known as Whole Slide Images (WSIs), may hamper the overall performance of CPATH systems. Deep Learning (DL) models such as Vision Transformers (ViTs) may detect and exclude artifacts before running the diagnostic algorithm. A simple way to develop robust and generalized ViTs is to train them on massive datasets. Unfortunately, acquiring large medical datasets is expensive and inconvenient, prompting the need for a generalized artifact detection method for WSIs. In this paper, we present a student-teacher recipe to improve the classification performance of ViT for the air bubbles detection task. ViT, trained under the student-teacher framework, boosts its performance by distilling existing knowledge from the high-capacity teacher model. Our best-performing ViT yields 0.961 and 0.911 F1-score and MCC, respectively, observing a 7% gain in MCC against stand-alone training. The proposed method presents a new perspective of leveraging knowledge distillation over transfer learning to encourage the use of customized transformers for efficient preprocessing pipelines in the CPATH systems.