# More From Less: Self-Supervised Knowledge Distillation for Routine Histopathology Data

arXiv: https://arxiv.org/abs/2303.10656 · HTML: https://mlsanity.com/paper/2303.10656/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [CrossTransformers: spatially-aware few-shot transfer](https://mlsanity.com/paper/2007.11498/) (2020)
- [Lizard: A Large-Scale Dataset for Colonic Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2108.11195/) (2021)
- [CoNIC: Colon Nuclei Identification and Counting Challenge 2022](https://mlsanity.com/paper/2111.14485/) (2021)
- [Resolution-Based Distillation for Efficient Histology Image Classification](https://mlsanity.com/paper/2101.04170/) (2021)
- [Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations](https://mlsanity.com/paper/2209.14905/) (2022)

## Built on by (most cited first, 1 of 1)

- [Divide and Conquer Self-Supervised Learning for High-Content Imaging](https://mlsanity.com/paper/2503.07444/) (2025)

## Abstract (arXiv)

Medical imaging technologies are generating increasingly large amounts of high-quality, information-dense data. Despite the progress, practical use of advanced imaging technologies for research and diagnosis remains limited by cost and availability, so information-sparse data such as H&E stains are relied on in practice. The study of diseased tissue requires methods which can leverage these information-dense data to extract more value from routine, information-sparse data. Using self-supervised deep learning, we demonstrate that it is possible to distil knowledge during training from information-dense data into models which only require information-sparse data for inference. This improves downstream classification accuracy on information-sparse data, making it comparable with the fully-supervised baseline. We find substantial effects on the learned representations, and this training process identifies subtle features which otherwise go undetected. This approach enables the design of models which require only routine images, but contain insights from state-of-the-art data, allowing better use of the available resources.