mlsanityData to 14 Oct 2026

Low-Rank Knowledge Decomposition for Medical Foundation Models

arXiv 2404.17184 ↗First posted 26 Apr 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

The popularity of large-scale pre-training has promoted the development of medical foundation models. However, some studies have shown that although foundation models exhibit strong general feature extraction capabilities, their performance on specific tasks is still inferior to task-specific methods. In this paper, we explore a new perspective called ``Knowledge Decomposition'' to improve the performance on specific medical tasks, which deconstruct the foundation model into multiple lightweight expert models, each dedicated to a particular task, with the goal of improving specialization while concurrently mitigating resource expenditure. To accomplish the above objective, we design a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates graidents by incorporating low-rank expert modules and the efficient knowledge separation convolution. Extensive experimental results demonstrate that the decomposed models perform well in terms of performance and transferability, even surpassing the original foundation models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 39 of its 67 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

39 of 67 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
27 more
  1. 2019Relational Knowledge Distillationcited by 889
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  3. 2017A Survey on Multi-Task Learningcited by 840
  4. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  5. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  6. 2019Contrastive Representation Distillationcited by 748
  7. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  9. 2018Born Again Neural Networkscited by 546
  10. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  11. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  12. 2019Similarity-Preserving Knowledge Distillationcited by 516
  13. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  14. 2019Variational Information Distillation for Knowledge Transfercited by 324
  15. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 239
  16. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  17. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  18. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  19. 2020COVID-CT-Dataset: A CT Scan Dataset about COVID-19cited by 94
  20. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  21. 2023Independent Component Alignment for Multi-Task Learningcited by 70
  22. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  23. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  24. 2022Factorizing Knowledge in Neural Networkscited by 59
  25. 2020ChestX-Det10: Chest X-ray Dataset on Detection of Thoracic Abnormalitiescited by 38
  26. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  27. 2022Risk of Bias in Chest Radiography Deep Learning Foundation Modelscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 2.

5 papers.

  1. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  2. 2024Reprogramming Distillation for Medical Foundation Modelscited by 5
  3. 2024Federated Learning under Partially Class-Disjoint Data via Manifold Reshapingcited by 4
  4. 2026Deep Reprogramming Distillation for Medical Foundation Modelscited by 0
  5. 2026Applying Language Models in Clinical Medicine: Recent Trends and Perspectivescited by 0