mlsanityData to 7 Oct 2026

Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models

arXiv 2607.05533 ↗First posted 6 Jul 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Computational pathology foundation models (PFMs) have advanced whole-slide image analysis. However, their size and inference cost hinder local deployment in pathology departments. We propose MuCoDi, a pretraining framework that distills frozen tile embeddings from multiple PFMs into compact edge-oriented encoders. Instead of regressing individual teacher features, MuCoDi trains lightweight MobileOne and RepViT students with a contrastive distillation objective adapted from MoCo v3, where cached Virchow2, UNI2, and H-Optimus-1 embeddings replace momentum-encoder keys. We pretrain students on 14.3M TCGA tiles from only 11.8K WSIs and evaluate frozen encoders on 23 clinically curated downstream classification tasks. RepViT-based MuCoEdge students retain near-teacher performance while reducing model size by orders of magnitude: MuCoEdge-R2.3 and MuCoEdge-R1.5 reach 71.0% external AUROC, within 0.8 percentage points of the best teacher (Virchow2, 71.8%), while MuCoEdge-R2.3 obtains the best external F1 and the second-best AUPRC (51.8% and 53.3%). MuCoEdge-R1.0 reaches 70.9% AUROC with only 6.4M parameters and 1.12 GFLOPs. On a Raspberry Pi 5, sub-million-parameter MobileOne students achieve up to 605-fold single-tile speedup over Virchow2 while retaining 66.5% to 66.9% external AUROC, demonstrating that PFM-quality pathology representations can be moved toward practical edge deployment. Code is available at https://anonymous.4open.science/r/mucodi-6243.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

21 of 37 references.

References · 21

  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2015Variational Inference with Normalizing Flowscited by 2,161
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  6. 2015Generating Sentences from a Continuous Spacecited by 607
  7. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  8. 2015Importance Weighted Autoencoderscited by 454
  9. 2017Wasserstein Auto-Encoderscited by 371
  10. 2017VAE with a VampPriorcited by 247
  11. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  12. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
9 more
  1. 2016Variational Lossy Autoencodercited by 175
  2. 2013Expectation Propagation for approximate Bayesian inferencecited by 172
  3. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  4. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  5. 2017Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networkscited by 107
  6. 2019Preventing Posterior Collapse with delta-VAEscited by 86
  7. 2016PixelVAE: A Latent Variable Model for Natural Imagescited by 69
  8. 2018Distribution Matching in Variational Inferencecited by 35
  9. 2023Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealingcited by 12

Built on by

Not cited yet.