# Reprogramming Distillation for Medical Foundation Models

arXiv: https://arxiv.org/abs/2407.06504 · HTML: https://mlsanity.com/paper/2407.06504/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [COVID-19 Image Data Collection](https://mlsanity.com/paper/2003.11597/) (2020)
- [COVID-CT-Dataset: A CT Scan Dataset about COVID-19](https://mlsanity.com/paper/2003.13865/) (2020)
- [Parameter-efficient Model Adaptation for Vision Transformers](https://mlsanity.com/paper/2203.16329/) (2022)
- [LPT: Long-tailed Prompt Tuning for Image Classification](https://mlsanity.com/paper/2210.01033/) (2022)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching](https://mlsanity.com/paper/2306.11925/) (2023)
- [Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective](https://mlsanity.com/paper/2203.08124/) (2022)
- [Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2212.03220/) (2022)
- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosis](https://mlsanity.com/paper/2311.08236/) (2023)
- [Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters](https://mlsanity.com/paper/2406.09679/) (2024)
- [Towards Efficient Task-Driven Model Reprogramming with Foundation Models](https://mlsanity.com/paper/2304.02263/) (2023)

## Built on by (most cited first, 5 of 5)

- [LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models](https://mlsanity.com/paper/2409.19540/) (2024)
- [RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis](https://mlsanity.com/paper/2509.19980/) (2025)
- [UniCAD: Efficient and Extendable Architecture for Multi-Task Computer-Aided Diagnosis System](https://mlsanity.com/paper/2505.09178/) (2025)
- [Deep Reprogramming Distillation for Medical Foundation Models](https://mlsanity.com/paper/2605.04447/) (2026)
- [Dementia Etiology Diagnosis via Collaborative Meta Knowledge Enhancement](https://mlsanity.com/paper/2607.22770/) (2026)

## Abstract (arXiv)

Medical foundation models pre-trained on large-scale datasets have demonstrated powerful versatile capabilities for various tasks. However, due to the gap between pre-training tasks (or modalities) and downstream tasks (or modalities), the real-world computation and speed constraints, it might not be straightforward to apply medical foundation models in the downstream scenarios. Previous methods, such as parameter efficient fine-tuning (PEFT) methods and knowledge distillation (KD) methods, are unable to simultaneously address the task (or modality) inconsistency and achieve personalized lightweight deployment under diverse real-world demands. To address the above issues, we propose a novel framework called Reprogramming Distillation (RD). On one hand, RD reprograms the original feature space of the foundation model so that it is more relevant to downstream scenarios, aligning tasks and modalities. On the other hand, through a co-training mechanism and a shared classifier, connections are established between the reprogrammed knowledge and the knowledge of student models, ensuring that the reprogrammed feature space can be smoothly mimic by the student model of different structures. Further, to reduce the randomness under different training conditions, we design a Centered Kernel Alignment (CKA) distillation to promote robust knowledge transfer. Empirically, we show that on extensive datasets, RD consistently achieve superior performance compared with previous PEFT and KD methods.