# Distilling foundation models for robust and efficient models in digital pathology

arXiv: https://arxiv.org/abs/2501.16239 · HTML: https://mlsanity.com/paper/2501.16239/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology](https://mlsanity.com/paper/2408.00738/) (2024)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis](https://mlsanity.com/paper/2406.16192/) (2024)
- [Phikon-v2, A large and public feature extractor for biomarker prediction](https://mlsanity.com/paper/2409.09173/) (2024)
- [Hibou: A Family of Foundational Vision Transformers for Pathology](https://mlsanity.com/paper/2406.05074/) (2024)
- [Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology](https://mlsanity.com/paper/2012.03583/) (2020)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [RudolfV: A Foundation Model by Pathologists for Pathologists](https://mlsanity.com/paper/2401.04079/) (2024)
- [Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation](https://mlsanity.com/paper/2407.18449/) (2024)
- [Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers](https://mlsanity.com/paper/2109.05819/) (2021)
- [Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios](https://mlsanity.com/paper/2410.16038/) (2024)
- [Atlas: A Novel Pathology Foundation Model by Mayo Clinic, Charit'e, and Aignostics](https://mlsanity.com/paper/2501.05409/) (2025)
- [A Simple Recipe for Competitive Low-compute Self supervised Vision Models](https://mlsanity.com/paper/2301.09451/) (2023)
- [Benchmarking Pathology Feature Extractors for Whole Slide Image Classification](https://mlsanity.com/paper/2311.11772/) (2023)
- [Benchmarking Domain Generalization Algorithms in Computational Pathology](https://mlsanity.com/paper/2409.17063/) (2024)

## Built on by (most cited first, 30 of 36)

- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [Benchmarking Deep Learning and Vision Foundation Models for Atypical vs. Normal Mitosis Classification with Cross-Dataset Evaluation](https://mlsanity.com/paper/2506.21444/) (2025)
- [Atlas 2 -- Foundation models for clinical deployment](https://mlsanity.com/paper/2601.05148/) (2026)
- [Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings](https://mlsanity.com/paper/2511.23204/) (2025)
- [Ensemble learning of pathology foundation models for precision oncology](https://mlsanity.com/paper/2508.16085/) (2025)
- [MOOZY: A Patient-First Foundation Model for Computational Pathology](https://mlsanity.com/paper/2603.27048/) (2026)
- [A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction](https://mlsanity.com/paper/2604.03630/) (2026)
- [Wasserstein Distances Made Explainable: Insights Into Dataset Shifts and Transport Phenomena](https://mlsanity.com/paper/2505.06123/) (2025)
- [Benchmarking Foundation Models for Mitotic Figure Classification](https://mlsanity.com/paper/2508.04441/) (2025)
- [Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology](https://mlsanity.com/paper/2603.08328/) (2026)
- [Unifying Multiple Foundation Models for Advanced Computational Pathology](https://mlsanity.com/paper/2503.00736/) (2025)
- [IMILIA: interpretable multiple instance learning for inflammation prediction in IBD from H&E whole slide images](https://mlsanity.com/paper/2512.13440/) (2025)
- [A Multicenter Benchmark of Multiple Instance Learning Models for Lymphoma Subtyping from HE-stained Whole Slide Images](https://mlsanity.com/paper/2512.14640/) (2025)
- [MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models](https://mlsanity.com/paper/2603.07895/) (2026)
- [CytoSyn: a Foundation Diffusion Model for Histopathology -- Tech Report](https://mlsanity.com/paper/2603.18089/) (2026)
- [Benchmarking Pathology Foundation Models for Breast Cancer Survival Prediction](https://mlsanity.com/paper/2604.24679/) (2026)
- [DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance](https://mlsanity.com/paper/2608.17872/) (2026)
- [Rank-Aware Agglomeration of Foundation Models for Immunohistochemistry Image Cell Counting](https://mlsanity.com/paper/2511.12588/) (2025)
- [SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classification](https://mlsanity.com/paper/2604.15711/) (2026)
- [A Statistical Multi-Objective Framework for Assessing Sensitivity of Radiomic AI Models to Acquisition Parameters](https://mlsanity.com/paper/2605.14667/) (2026)
- [Do Foundation Models See Biology? Evaluating Attention Coherence with Spatial Transcriptomics in Glioblastoma](https://mlsanity.com/paper/2606.04764/) (2026)
- [LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification](https://mlsanity.com/paper/2606.06864/) (2026)
- [From Patches to Patients: A study of the tile-to-slide performance transferability in Digital Pathology](https://mlsanity.com/paper/2606.10778/) (2026)
- [Higher-Order Certified Robustness for Regression](https://mlsanity.com/paper/2607.05536/) (2026)
- [LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models](https://mlsanity.com/paper/2607.11257/) (2026)
- [Robustifying pathology foundation models via fine-tuning](https://mlsanity.com/paper/2607.22861/) (2026)
- [A Distributional Robustness Margin For Pathology Foundation Models](https://mlsanity.com/paper/2607.25497/) (2026)
- [Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast Cancer](https://mlsanity.com/paper/2608.01356/) (2026)
- [Bag-of-Visual-Words for Spatial Mapping of Lung Adenocarcinoma Growth Patterns](https://mlsanity.com/paper/2608.05074/) (2026)
- [P3CA: Encoder-Agnostic Interpretation of Vision Foundation Model Embeddings via Spatial Probing](https://mlsanity.com/paper/2608.10131/) (2026)

## Abstract (arXiv)

In recent years, the advent of foundation models (FM) for digital pathology has relied heavily on scaling the pre-training datasets and the model size, yielding large and powerful models. While it resulted in improving the performance on diverse downstream tasks, it also introduced increased computational cost and inference time. In this work, we explore the distillation of a large foundation model into a smaller one, reducing the number of parameters by several orders of magnitude. Leveraging distillation techniques, our distilled model, H0-mini, achieves nearly comparable performance to large FMs at a significantly reduced inference cost. It is evaluated on several public benchmarks, achieving 3rd place on the HEST benchmark and 5th place on the EVA benchmark. Additionally, a robustness analysis conducted on the PLISM dataset demonstrates that our distilled model reaches excellent robustness to variations in staining and scanning conditions, significantly outperforming other state-of-the art models. This opens new perspectives to design lightweight and robust models for digital pathology, without compromising on performance.