# Foundation X: Integrating Classification, Localization, and Segmentation through Lock-Release Pretraining Strategy for Chest X-ray Analysis

arXiv: https://arxiv.org/abs/2503.09860 · HTML: https://mlsanity.com/paper/2503.09860/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [Multi-Task Self-Training for Learning General Representations](https://mlsanity.com/paper/2108.11353/) (2021)
- [A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation](https://mlsanity.com/paper/2104.10326/) (2021)
- [CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images](https://mlsanity.com/paper/2307.03293/) (2023)
- [VinDr-RibCXR: A Benchmark Dataset for Automatic Segmentation and Labeling of Individual Ribs on Chest X-rays](https://mlsanity.com/paper/2107.01327/) (2021)
- [Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation](https://mlsanity.com/paper/2311.01092/) (2023)
- [Foundation Ark: Accruing and Reusing Knowledge for Superior and Robust Performance](https://mlsanity.com/paper/2310.09507/) (2023)
- [Multi-Task Driven Explainable Diagnosis of COVID-19 using Chest X-ray Images](https://mlsanity.com/paper/2008.03205/) (2020)
- [Nodule detection and generation on chest X-rays: NODE21 Challenge](https://mlsanity.com/paper/2401.02192/) (2024)
- [X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation](https://mlsanity.com/paper/2203.08764/) (2022)
- [Universal Object Detection with Large Vision Model](https://mlsanity.com/paper/2212.09408/) (2022)
- [ViM: Vision Middleware for Unified Downstream Transferring](https://mlsanity.com/paper/2303.06911/) (2023)

## Built on by (most cited first, 3 of 3)

- [Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays](https://mlsanity.com/paper/2603.26049/) (2026)
- [Analogical Reasoning as a Doctor: A Foundation Model for Gastrointestinal Endoscopy Diagnosis](https://mlsanity.com/paper/2604.05649/) (2026)
- [Beyond Masks: The Case for Medical Image Parsing](https://mlsanity.com/paper/2605.11438/) (2026)

## Abstract (arXiv)

Developing robust and versatile deep-learning models is essential for enhancing diagnostic accuracy and guiding clinical interventions in medical imaging, but it requires a large amount of annotated data. The advancement of deep learning has facilitated the creation of numerous medical datasets with diverse expert-level annotations. Aggregating these datasets can maximize data utilization and address the inadequacy of labeled data. However, the heterogeneity of expert-level annotations across tasks such as classification, localization, and segmentation presents a significant challenge for learning from these datasets. To this end, we introduce nFoundation X, an end-to-end framework that utilizes diverse expert-level annotations from numerous public datasets to train a foundation model capable of multiple tasks including classification, localization, and segmentation. To address the challenges of annotation and task heterogeneity, we propose a Lock-Release pretraining strategy to enhance the cyclic learning from multiple datasets, combined with the student-teacher learning paradigm, ensuring the model retains general knowledge for all tasks while preventing overfitting to any single task. To demonstrate the effectiveness of Foundation X, we trained a model using 11 chest X-ray datasets, covering annotations for classification, localization, and segmentation tasks. Our experimental results show that Foundation X achieves notable performance gains through extensive annotation utilization, excels in cross-dataset and cross-task learning, and further enhances performance in organ localization and segmentation tasks. All code and pretrained models are publicly accessible at https://github.com/jlianglab/Foundation_X.