# MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classification

arXiv: https://arxiv.org/abs/2306.09579 · HTML: https://mlsanity.com/paper/2306.09579/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning](https://mlsanity.com/paper/1602.03409/) (2016)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Self-Supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotation](https://mlsanity.com/paper/2007.09886/) (2020)
- [Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study](https://mlsanity.com/paper/2209.15517/) (2022)
- [Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding](https://mlsanity.com/paper/2012.05440/) (2020)
- [FHIST: A Benchmark for Few-shot Classification of Histological Images](https://mlsanity.com/paper/2206.00092/) (2022)

## Built on by (most cited first, 30 of 38)

- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models](https://mlsanity.com/paper/2407.10990/) (2024)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [CheXWorld: Exploring Image World Modeling for Radiograph Representation Learning](https://mlsanity.com/paper/2504.13820/) (2025)
- [PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion](https://mlsanity.com/paper/2503.00196/) (2025)
- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [General-purpose foundation models for increased autonomy in robot-assisted surgery](https://mlsanity.com/paper/2401.00678/) (2024)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification](https://mlsanity.com/paper/2512.12887/) (2025)
- [Embedded Visual Prompt Tuning](https://mlsanity.com/paper/2407.01003/) (2024)
- [Federated Learning for Medical Image Classification: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.05238/) (2025)
- [OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine](https://mlsanity.com/paper/2402.18028/) (2024)
- [Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2410.21302/) (2024)
- [Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes](https://mlsanity.com/paper/2507.00327/) (2025)
- [MedFormer: Hierarchical Medical Vision Transformer with Content-Aware Dual Sparse Selection Attention](https://mlsanity.com/paper/2507.02488/) (2025)
- [Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language](https://mlsanity.com/paper/2603.02790/) (2026)
- [MambaMIC: An Efficient Baseline for Microscopic Image Classification with State Space Models](https://mlsanity.com/paper/2409.07896/) (2024)
- [Exploring Representation Invariance in Finetuning](https://mlsanity.com/paper/2503.07399/) (2025)
- [Pre-trained Models Succeed in Medical Imaging with Representation Similarity Degradation](https://mlsanity.com/paper/2503.07958/) (2025)
- [A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling](https://mlsanity.com/paper/2602.22843/) (2026)
- [VR-FuseNet: A Fusion of Heterogeneous Fundus Data and Explainable Deep Network for Diabetic Retinopathy Classification](https://mlsanity.com/paper/2504.21464/) (2025)
- [Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning](https://mlsanity.com/paper/2512.03667/) (2025)
- [Learning Longitudinal Health Representations from EHR and Wearable Data](https://mlsanity.com/paper/2601.12227/) (2026)
- [A Hierarchical Benchmark of Foundation Models for Dermatology](https://mlsanity.com/paper/2601.12382/) (2026)
- [Baichuan-M4: A Clinical-Grade Medical Agent System for Continuous Care](https://mlsanity.com/paper/2606.08982/) (2026)
- [Downstream Analysis of Foundational Medical Vision Models for Disease Progression](https://mlsanity.com/paper/2503.16842/) (2025)
- [Diagnostic Accuracy of Open-Source Vision-Language Models on Diverse Medical Imaging Tasks](https://mlsanity.com/paper/2508.01016/) (2025)
- [Information Bottleneck-based Causal Attention for Multi-label Medical Image Recognition](https://mlsanity.com/paper/2508.08069/) (2025)

## Abstract (arXiv)

Foundation models, often pre-trained with large-scale data, have achieved paramount success in jump-starting various vision and language applications. Recent advances further enable adapting foundation models in downstream tasks efficiently using only a few training samples, e.g., in-context learning. Yet, the application of such learning paradigms in medical image analysis remains scarce due to the shortage of publicly accessible data and benchmarks. In this paper, we aim at approaches adapting the foundation models for medical image classification and present a novel dataset and benchmark for the evaluation, i.e., examining the overall performance of accommodating the large-scale foundation models downstream on a set of diverse real-world clinical tasks. We collect five sets of medical imaging data from multiple institutes targeting a variety of real-world clinical tasks (22,349 images in total), i.e., thoracic diseases screening in X-rays, pathological lesion tissue screening, lesion detection in endoscopy images, neonatal jaundice evaluation, and diabetic retinopathy grading. Results of multiple baseline methods are demonstrated using the proposed dataset from both accuracy and cost-effective perspectives.