# Overcoming Data Scarcity in Biomedical Imaging with a Foundational Multi-Task Model

arXiv: https://arxiv.org/abs/2311.09847 · HTML: https://mlsanity.com/paper/2311.09847/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 77 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [A survey of loss functions for semantic segmentation](https://mlsanity.com/paper/2006.14822/) (2020)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [FCOS: A simple and strong anchor-free object detector](https://mlsanity.com/paper/2006.09214/) (2020)

## Built on by (most cited first, 30 of 40)

- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models](https://mlsanity.com/paper/2403.14066/) (2024)
- [Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception](https://mlsanity.com/paper/2509.15333/) (2025)
- [Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimes](https://mlsanity.com/paper/2408.17421/) (2024)
- [Tissue Concepts: supervised foundation models in computational pathology](https://mlsanity.com/paper/2409.03519/) (2024)
- [Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation](https://mlsanity.com/paper/2508.09177/) (2025)
- [HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models](https://mlsanity.com/paper/2405.07460/) (2024)
- [CoLA: Collaborative Low-Rank Adaptation](https://mlsanity.com/paper/2505.15471/) (2025)
- [Meningioma Analysis and Diagnosis using Limited Labeled Samples](https://mlsanity.com/paper/2602.13335/) (2026)
- [Translating Electrocardiograms to Cardiac Magnetic Resonance Imaging Useful for Cardiac Assessment and Disease Screening: A Multi-Center Study](https://mlsanity.com/paper/2411.13602/) (2024)
- [Foundation Model for Whole-Heart Segmentation: Leveraging Student-Teacher Learning in Multi-Modal Medical Imaging](https://mlsanity.com/paper/2503.19005/) (2025)
- [Foundation Models in Biomedical Imaging: Turning Hype into Reality](https://mlsanity.com/paper/2512.15808/) (2025)
- [Interpreting Biomedical VLMs on High-Imbalance Out-of-Distributions: An Insight into BiomedCLIP on Radiology](https://mlsanity.com/paper/2506.14136/) (2025)
- [Whole Slide Concepts: A Supervised Foundation Model For Pathological Images](https://mlsanity.com/paper/2507.05742/) (2025)
- [MAPLE: Multi-scale Attribute-enhanced Prompt Learning for Few-shot Whole Slide Image Classification](https://mlsanity.com/paper/2509.25863/) (2025)
- [ReaMIL: Reasoning- and Evidence-Aware Multiple Instance Learning for Whole-Slide Histopathology](https://mlsanity.com/paper/2601.10073/) (2026)
- [Stylizing ViT: Anatomy-Preserving Instance Style Transfer for Domain Generalization](https://mlsanity.com/paper/2601.17586/) (2026)
- [Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention](https://mlsanity.com/paper/2605.06699/) (2026)
- [Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation](https://mlsanity.com/paper/2607.07219/) (2026)
- [Latent Communication Between Language Model Agents: Channels, Alignment, and the Limits of Text](https://mlsanity.com/paper/2607.14103/) (2026)
- [Architecture independent generalization bounds for overparametrized deep ReLU networks](https://mlsanity.com/paper/2504.05695/) (2025)
- [Explainable Anatomy-Guided AI for Prostate MRI: Foundation Models and In Silico Clinical Trials for Virtual Biopsy-based Risk Assessment](https://mlsanity.com/paper/2505.17971/) (2025)
- [A Pre-trained Foundation Model Framework for Multiplanar MRI Classification of Extramural Vascular Invasion and Mesorectal Fascia Invasion in Rectal Cancer](https://mlsanity.com/paper/2505.18058/) (2025)
- [ALL-PET: A Low-resource and Low-shot PET Foundation Model in Projection Domain](https://mlsanity.com/paper/2509.09130/) (2025)
- [DuPLUS: Dual-Prompt Vision-Language Framework for Universal Medical Image Segmentation and Prognosis](https://mlsanity.com/paper/2510.03483/) (2025)
- [CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging](https://mlsanity.com/paper/2511.11034/) (2025)
- [Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization](https://mlsanity.com/paper/2512.06006/) (2025)
- [EvalBlocks: A Modular Pipeline for Rapidly Evaluating Foundation Models in Medical Imaging](https://mlsanity.com/paper/2601.03811/) (2026)
- [Robust Multicentre Detection and Classification of Colorectal Liver Metastases on CT: Application of Foundation Models](https://mlsanity.com/paper/2601.07585/) (2026)
- [Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learning](https://mlsanity.com/paper/2603.00811/) (2026)

## Abstract (arXiv)

Foundational models, pretrained on a large scale, have demonstrated substantial success across non-medical domains. However, training these models typically requires large, comprehensive datasets, which contrasts with the smaller and more heterogeneous datasets common in biomedical imaging. Here, we propose a multi-task learning strategy that decouples the number of training tasks from memory requirements. We trained a Universal bioMedical PreTrained model (UMedPT) on a multi-task database including tomographic, microscopic, and X-ray images, with various labelling strategies such as classification, segmentation, and object detection. The UMedPT foundational model outperformed ImageNet pretraining and the previous state-of-the-art models. For tasks related to the pretraining database, it maintained its performance with only 1% of the original training data and without fine-tuning. For out-of-domain tasks it required not more than 50% of the original training data. In an external independent validation imaging features extracted using UMedPT proved to be a new standard for cross-center transferability.