# Resolution-Based Distillation for Efficient Histology Image Classification

arXiv: https://arxiv.org/abs/2101.04170 · HTML: https://mlsanity.com/paper/2101.04170/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 65 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Representation Learning by Learning to Count](https://mlsanity.com/paper/1708.06734/) (2017)
- [Knowledge Adaptation for Efficient Semantic Segmentation](https://mlsanity.com/paper/1903.04688/) (2019)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation](https://mlsanity.com/paper/1811.09998/) (2018)
- [Detecting Cancer Metastases on Gigapixel Pathology Images](https://mlsanity.com/paper/1703.02442/) (2017)
- [Weakly supervised multiple instance learning histopathological tumor segmentation](https://mlsanity.com/paper/2004.05024/) (2020)
- [Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks](https://mlsanity.com/paper/1901.11489/) (2019)
- [Dense Steerable Filter CNNs for Exploiting Rotational Symmetry in Histology Images](https://mlsanity.com/paper/2004.03037/) (2020)
- [Self-Supervised Nuclei Segmentation in Histopathological Images Using Attention](https://mlsanity.com/paper/2007.08373/) (2020)
- [Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides](https://mlsanity.com/paper/1811.08513/) (2018)
- [Processing Megapixel Images with Deep Attention-Sampling Models](https://mlsanity.com/paper/1905.03711/) (2019)
- [Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images](https://mlsanity.com/paper/1703.01550/) (2017)
- [Self-Supervised Similarity Learning for Digital Pathology](https://mlsanity.com/paper/1905.08139/) (2019)
- [Detection of prostate cancer in whole-slide images through end-to-end training with image-level labels](https://mlsanity.com/paper/2006.03394/) (2020)
- [Knowledge Distillation with Feature Maps for Image Classification](https://mlsanity.com/paper/1812.00660/) (2018)
- [Automated detection of celiac disease on duodenal biopsy slides: a deep learning approach](https://mlsanity.com/paper/1901.11447/) (2019)
- [Breast Cancer Histopathology Image Classification and Localization using Multiple Instance Learning](https://mlsanity.com/paper/2003.00823/) (2020)
- [Training convolutional neural networks with megapixel images](https://mlsanity.com/paper/1804.05712/) (2018)

## Built on by (most cited first, 10 of 10)

- [Masked Pre-Training of Transformers for Histology Image Analysis](https://mlsanity.com/paper/2304.07434/) (2023)
- [Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy](https://mlsanity.com/paper/2306.07999/) (2023)
- [Improving Representation Learning for Histopathologic Images with Cluster Constraints](https://mlsanity.com/paper/2310.12334/) (2023)
- [MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis](https://mlsanity.com/paper/2308.16561/) (2023)
- [Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.16464/) (2022)
- [Patch Gradient Descent: Training Neural Networks on Very Large Images](https://mlsanity.com/paper/2301.13817/) (2023)
- [Structured Model Pruning for Efficient Inference in Computational Pathology](https://mlsanity.com/paper/2404.08831/) (2024)
- [More From Less: Self-Supervised Knowledge Distillation for Routine Histopathology Data](https://mlsanity.com/paper/2303.10656/) (2023)
- [A Deep Reinforcement Learning Framework for Rapid Diagnosis of Whole Slide Pathological Images](https://mlsanity.com/paper/2205.02850/) (2022)
- [Attention to detail: inter-resolution knowledge distillation](https://mlsanity.com/paper/2401.06010/) (2024)

## Abstract (arXiv)

Developing deep learning models to analyze histology images has been computationally challenging, as the massive size of the images causes excessive strain on all parts of the computing pipeline. This paper proposes a novel deep learning-based methodology for improving the computational efficiency of histology image classification. The proposed approach is robust when used with images that have reduced input resolution and can be trained effectively with limited labeled data. Pre-trained on the original high-resolution (HR) images, our method uses knowledge distillation (KD) to transfer learned knowledge from a teacher model to a student model trained on the same images at a much lower resolution. To address the lack of large-scale labeled histology image datasets, we perform KD in a self-supervised manner. We evaluate our approach on two histology image datasets associated with celiac disease (CD) and lung adenocarcinoma (LUAD). Our results show that a combination of KD and self-supervision allows the student model to approach, and in some cases, surpass the classification accuracy of the teacher, while being much more efficient. Additionally, we observe an increase in student classification performance as the size of the unlabeled dataset increases, indicating that there is potential to scale further. For the CD data, our model outperforms the HR teacher model, while needing 4 times fewer computations. For the LUAD data, our student model results at 1.25x magnification are within 3% of the teacher model at 10x magnification, with a 64 times computational cost reduction. Moreover, our CD outcomes benefit from performance scaling with the use of more unlabeled data. For 0.625x magnification, using unlabeled data improves accuracy by 4% over the baseline. Thus, our method can improve the feasibility of deep learning solutions for digital pathology with standard computational hardware.