# DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance

arXiv: https://arxiv.org/abs/2608.17872 · HTML: https://mlsanity.com/paper/2608.17872/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 18 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology](https://mlsanity.com/paper/2408.00738/) (2024)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis](https://mlsanity.com/paper/2406.16192/) (2024)
- [Phikon-v2, A large and public feature extractor for biomarker prediction](https://mlsanity.com/paper/2409.09173/) (2024)
- [A Petri Dish for Histopathology Image Analysis](https://mlsanity.com/paper/2101.12355/) (2021)
- [Distilling foundation models for robust and efficient models in digital pathology](https://mlsanity.com/paper/2501.16239/) (2025)
- [Towards Large-Scale Training of Pathology Foundation Models](https://mlsanity.com/paper/2404.15217/) (2024)
- [Training state-of-the-art pathology foundation models with orders of magnitude less data](https://mlsanity.com/paper/2504.05186/) (2025)
- [A Simple Recipe for Competitive Low-compute Self supervised Vision Models](https://mlsanity.com/paper/2301.09451/) (2023)
- [PLUTO-4: Frontier Pathology Foundation Models](https://mlsanity.com/paper/2511.02826/) (2025)

## Built on by (most cited first, 1 of 1)

- [One-Slide Calibration of Pathology Foundation Models](https://mlsanity.com/paper/2610.08944/) (2026)

## Abstract (arXiv)

Many high-performing pathology tile encoders are now foundation models with hundreds of millions to over a billion parameters. Encoding and storing the thousands of tiles in each whole-slide image with such models is costly on commodity hardware, so compact encoders that retain useful downstream performance are a valuable alternative. We present DistillPath-KS16, which starts from the existing 22M kaiko ViT-S/16 encoder and improves it by distilling from released pathology encoders used as frozen teachers. The recipe reads only the teachers' final class and patch tokens and trains on 6,000 public slides, needing neither their DINO nor iBOT pretraining heads nor a billion-tile corpus, so it applies to any released encoder that exposes backbone tokens. We distill four teachers spanning 86M to 1.1B parameters into the same student. Every variant improves the kaiko baseline on all three benchmarks we use, EVA, HEST, and PLISM, and the strongest teacher is task-dependent. On the seven-task EVA mean, DistillPath-KS16-Virchow2 reaches 0.795, within 0.015 points of Virchow2, the top-scoring model in our evaluation, at about 29× fewer parameters; it also scores above H0-mini and GPFM on this aggregate metric, though that advantage is task-concentrated rather than uniform. Because it remains a 22M ViT-S/16 with 384-dimensional features, DistillPath-KS16 runs more than 25× faster than Virchow2. Code is available at https://github.com/RamonKaspar/DistillPath, and released model weights are available at https://huggingface.co/collections/RamonK/distillpath.