mlsanityData to 7 Oct 2026

Distilling foundation models for robust and efficient models in digital pathology

arXiv 2501.16239 ↗First posted 27 Jan 2025cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

In recent years, the advent of foundation models (FM) for digital pathology has relied heavily on scaling the pre-training datasets and the model size, yielding large and powerful models. While it resulted in improving the performance on diverse downstream tasks, it also introduced increased computational cost and inference time. In this work, we explore the distillation of a large foundation model into a smaller one, reducing the number of parameters by several orders of magnitude. Leveraging distillation techniques, our distilled model, H0-mini, achieves nearly comparable performance to large FMs at a significantly reduced inference cost. It is evaluated on several public benchmarks, achieving 3rd place on the HEST benchmark and 5th place on the EVA benchmark. Additionally, a robustness analysis conducted on the PLISM dataset demonstrates that our distilled model reaches excellent robustness to variations in staining and scanning conditions, significantly outperforming other state-of-the art models. This opens new perspectives to design lightweight and robust models for digital pathology, without compromising on performance.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

25 of 47 references.

References · 25

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  8. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  9. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 254
  10. 2020Self supervised contrastive learning for digital histopathologycited by 146
  11. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  12. 2023Artificial Intelligence for Digital and Computational Pathologycited by 106
13 more
  1. 2024HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysiscited by 104
  2. 2024Phikon-v2, A large and public feature extractor for biomarker predictioncited by 67
  3. 2024Hibou: A Family of Foundational Vision Transformers for Pathologycited by 52
  4. 2020Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histologycited by 43
  5. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  6. 2024RudolfV: A Foundation Model by Pathologists for Pathologistscited by 40
  7. 2024Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillationcited by 30
  8. 2021Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancerscited by 23
  9. 2024Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarioscited by 16
  10. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Modelscited by 10
  11. 2025Atlas: A Novel Pathology Foundation Model by Mayo Clinic, Charit'e, and Aignosticscited by 10
  12. 2023Benchmarking Pathology Feature Extractors for Whole Slide Image Classificationcited by 9
  13. 2024Benchmarking Domain Generalization Algorithms in Computational Pathologycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 23.

33 papers.

Sort:

By area:

  1. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 7
  2. 2025Benchmarking Deep Learning and Vision Foundation Models for Atypical vs. Normal Mitosis Classification with Cross-Dataset Evaluationcited by 7
  3. 2026Atlas 2 -- Foundation models for clinical deploymentcited by 6
  4. 2025Ensemble learning of pathology foundation models for precision oncologycited by 4
  5. 2025Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddingscited by 4
  6. 2026A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Predictioncited by 3
  7. 2025Wasserstein Distances Made Explainable: Insights Into Dataset Shifts and Transport Phenomenacited by 2
  8. 2025Benchmarking Foundation Models for Mitotic Figure Classificationcited by 2
  9. 2026Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathologycited by 2
  10. 2025Unifying Multiple Foundation Models for Advanced Computational Pathologycited by 1
  11. 2025IMILIA: interpretable multiple instance learning for inflammation prediction in IBD from H&E whole slide imagescited by 1
  12. 2025A Multicenter Benchmark of Multiple Instance Learning Models for Lymphoma Subtyping from HE-stained Whole Slide Imagescited by 1
21 more
  1. 2026MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Modelscited by 1
  2. 2026Benchmarking Pathology Foundation Models for Breast Cancer Survival Predictioncited by 1
  3. 2025Rank-Aware Agglomeration of Foundation Models for Immunohistochemistry Image Cell Countingcited by 0
  4. 2026CytoSyn: a Foundation Diffusion Model for Histopathology -- Tech Reportcited by 0
  5. 2026SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classificationcited by 0
  6. 2026A Statistical Multi-Objective Framework for Assessing Sensitivity of Radiomic AI Models to Acquisition Parameterscited by 0
  7. 2026Do Foundation Models See Biology? Evaluating Attention Coherence with Spatial Transcriptomics in Glioblastomacited by 0
  8. 2026LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classificationcited by 0
  9. 2026From Patches to Patients: A study of the tile-to-slide performance transferability in Digital Pathologycited by 0
  10. 2026Higher-Order Certified Robustness for Regressioncited by 0
  11. 2026LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Modelscited by 0
  12. 2026Robustifying pathology foundation models via fine-tuningcited by 0
  13. 2026A Distributional Robustness Margin For Pathology Foundation Modelscited by 0
  14. 2026Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast Cancercited by 0
  15. 2026Bag-of-Visual-Words for Spatial Mapping of Lung Adenocarcinoma Growth Patternscited by 0
  16. 2026P3CA: Encoder-Agnostic Interpretation of Vision Foundation Model Embeddings via Spatial Probingcited by 0
  17. 2026DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performancecited by 0
  18. 2026ADMIL: Attention-Distilled Multiple Instance Learning for Selective Foundation Model Inference in Pathologycited by 0
  19. 2026EXPOSE: Explainable and Domain-Robust Embeddings from Pathology Vision Foundation Models using Sparse Autoencoderscited by 0
  20. 2026Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathologycited by 0
  21. 2026HERO: Histology Encoder for Robust Representation in Oncologycited by 0