# UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classification

arXiv: https://arxiv.org/abs/2312.11038 · HTML: https://mlsanity.com/paper/2312.11038/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [ChestX-Det10: Chest X-ray Dataset on Detection of Thoracic Abnormalities](https://mlsanity.com/paper/2006.10550/) (2020)
- [On Multi-Domain Long-Tailed Recognition, Imbalanced Domain Generalization and Beyond](https://mlsanity.com/paper/2203.09513/) (2022)
- [Contrastive Language-Image Pre-training for the Italian Language](https://mlsanity.com/paper/2108.08688/) (2021)
- [Darwin's Neural Network: AI-based Strategies for Rapid and Scalable Cell and Coronavirus Screening](https://mlsanity.com/paper/2007.11653/) (2020)

## Built on by (most cited first, 21 of 21)

- [Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning](https://mlsanity.com/paper/2405.01469/) (2024)
- [Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging](https://mlsanity.com/paper/2403.04484/) (2024)
- [DeViDe: Faceted medical knowledge for improved medical vision-language pre-training](https://mlsanity.com/paper/2404.03618/) (2024)
- [CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray](https://mlsanity.com/paper/2506.07984/) (2025)
- [Long-tailed Recognition with Model Rebalancing](https://mlsanity.com/paper/2510.08177/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classification](https://mlsanity.com/paper/2406.05677/) (2024)
- [Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention](https://mlsanity.com/paper/2502.13693/) (2025)
- [Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models](https://mlsanity.com/paper/2506.08990/) (2025)
- [LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models](https://mlsanity.com/paper/2409.19540/) (2024)
- [Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation](https://mlsanity.com/paper/2411.04607/) (2024)
- [Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning](https://mlsanity.com/paper/2507.12998/) (2025)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment](https://mlsanity.com/paper/2403.10635/) (2024)
- [RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis](https://mlsanity.com/paper/2509.19980/) (2025)
- [Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Data](https://mlsanity.com/paper/2510.08179/) (2025)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions](https://mlsanity.com/paper/2608.07550/) (2026)
- [AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via α-Corrected Binary Cross Entropy and Factorized Latent Supervision](https://mlsanity.com/paper/2609.01757/) (2026)
- [Exponential Pixelating Integral transform with dual fractal features for enhanced chest X-ray abnormality detection](https://mlsanity.com/paper/2609.10988/) (2026)

## Abstract (arXiv)

Vision-Language Pre-training (VLP) that utilizes the multi-modal information to promote the training efficiency and effectiveness, has achieved great success in vision recognition of natural domains and shown promise in medical imaging diagnosis for the Chest X-Rays (CXRs). However, current works mainly pay attention to the exploration on single dataset of CXRs, which locks the potential of this powerful paradigm on larger hybrid of multi-source CXRs datasets. We identify that although blending samples from the diverse sources offers the advantages to improve the model generalization, it is still challenging to maintain the consistent superiority for the task of each source due to the existing heterogeneity among sources. To handle this dilemma, we design a Conquer-and-Divide pre-training framework, termed as UniChest, aiming to make full use of the collaboration benefit of multiple sources of CXRs while reducing the negative influence of the source heterogeneity. Specially, the ``Conquer" stage in UniChest encourages the model to sufficiently capture multi-source common patterns, and the ``Divide" stage helps squeeze personalized patterns into different small experts (query networks). We conduct thorough experiments on many benchmarks, e.g., ChestX-ray14, CheXpert, Vindr-CXR, Shenzhen, Open-I and SIIM-ACR Pneumothorax, verifying the effectiveness of UniChest over a range of baselines, and release our codes and pre-training models at https://github.com/Elfenreigen/UniChest.