# HistoTransfer: Understanding Transfer Learning for Histopathology

arXiv: https://arxiv.org/abs/2106.07068 · HTML: https://mlsanity.com/paper/2106.07068/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 13 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2103.10626/) (2021)
- [CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2101.06871/) (2021)
- [Multi-task pre-training of deep neural networks for digital pathology](https://mlsanity.com/paper/2005.02561/) (2020)

## Built on by (most cited first, 6 of 6)

- [Evaluating histopathology transfer learning with ChampKit](https://mlsanity.com/paper/2206.06862/) (2022)
- [Towards Launching AI Algorithms for Cellular Pathology into Clinical & Pharmaceutical Orbits](https://mlsanity.com/paper/2112.09496/) (2021)
- [Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Review](https://mlsanity.com/paper/2410.19820/) (2024)
- [Domain-specific transfer learning in the automated scoring of tumor-stroma ratio from histopathological images of colorectal cancer](https://mlsanity.com/paper/2212.14652/) (2022)
- [Leveraging Spatial Context for Positive Pair Sampling in Histopathology Image Representation Learning](https://mlsanity.com/paper/2503.05170/) (2025)
- [Overfitting in Histopathology Model Training: The Need for Customized Architectures](https://mlsanity.com/paper/2506.16631/) (2025)

## Abstract (arXiv)

Advancement in digital pathology and artificial intelligence has enabled deep learning-based computer vision techniques for automated disease diagnosis and prognosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized, making them infeasible to be used directly for training deep neural networks. Hence, for modeling, a two-stage approach is adopted: Patch representations are extracted first, followed by the aggregation for WSI prediction. These approaches require detailed pixel-level annotations for training the patch encoder. However, obtaining these annotations is time-consuming and tedious for medical experts. Transfer learning is used to address this gap and deep learning architectures pre-trained on ImageNet are used for generating patch-level representation. Even though ImageNet differs significantly from histopathology data, pre-trained networks have been shown to perform impressively on histopathology data. Also, progress in self-supervised and multi-task learning coupled with the release of multiple histopathology data has led to the release of histopathology-specific networks. In this work, we compare the performance of features extracted from networks trained on ImageNet and histopathology data. We use an attention pooling network over these extracted features for slide-level aggregation. We investigate if features learned using more complex networks lead to gain in performance. We use a simple top-k sampling approach for fine-tuning framework and study the representation similarity between frozen and fine-tuned networks using Centered Kernel Alignment. Further, to examine if intermediate block representation is better suited for feature extraction and ImageNet architectures are unnecessarily large for histopathology, we truncate the blocks of ResNet18 and DenseNet121 and examine the performance.