# Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models

arXiv: https://arxiv.org/abs/2404.04936 · HTML: https://mlsanity.com/paper/2404.04936/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 7 Apr 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [Contrastive Embedding for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2103.16173/) (2021)
- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)

## Built on by (most cited first, 24 of 24)

- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [Towards Scalable Language-Image Pre-training for 3D Medical Imaging](https://mlsanity.com/paper/2505.21862/) (2025)
- [Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training](https://mlsanity.com/paper/2508.03742/) (2025)
- [Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis](https://mlsanity.com/paper/2501.03565/) (2025)
- [More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era](https://mlsanity.com/paper/2509.13175/) (2025)
- [SimCroP: Radiograph Representation Learning with Similarity-driven Cross-granularity Pre-training](https://mlsanity.com/paper/2509.08311/) (2025)
- [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025)
- [Learning complete and explainable visual representations from itemized text supervision](https://mlsanity.com/paper/2512.11141/) (2025)
- [GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training](https://mlsanity.com/paper/2603.06467/) (2026)
- [Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysis](https://mlsanity.com/paper/2603.19516/) (2026)
- [Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge](https://mlsanity.com/paper/2607.10953/) (2026)
- [Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration](https://mlsanity.com/paper/2506.10573/) (2025)
- [Cardiac-CLIP: A Vision-Language Foundation Model for 3D Cardiac CT Images](https://mlsanity.com/paper/2507.22024/) (2025)
- [ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training](https://mlsanity.com/paper/2606.00602/) (2026)
- [Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography](https://mlsanity.com/paper/2606.25546/) (2026)
- [Anatomy Contextualized Adaptation of CT Foundation Models](https://mlsanity.com/paper/2607.27154/) (2026)
- [ARC-CT: Anatomy-Routed Contrastive Vision-Language Learning for 3D Chest CT](https://mlsanity.com/paper/2608.28455/) (2026)
- [3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection](https://mlsanity.com/paper/2602.23652/) (2026)
- [Unleashing Video Language Models for Fine-grained HRCT Report Generation](https://mlsanity.com/paper/2603.12469/) (2026)
- [Learning from Compressed CT: Feature Attention Style Transfer and Structured Factorized Projections for Resource-Efficient Medical Image Analysis](https://mlsanity.com/paper/2605.00448/) (2026)
- [Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors](https://mlsanity.com/paper/2606.23356/) (2026)
- [Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy](https://mlsanity.com/paper/2607.09135/) (2026)
- [Fine-Grained Vision-Language Pretraining with Organ-Conditioned Pattern Tokens for CT Understanding](https://mlsanity.com/paper/2607.13892/) (2026)
- [Semantically Calibrated Evidence Composition for CT Vision-Language Learning](https://mlsanity.com/paper/2608.00239/) (2026)

## Abstract (arXiv)

Radiologists highly desire fully automated versatile AI for medical imaging interpretation. However, the lack of extensively annotated large-scale multi-disease datasets has hindered the achievement of this goal. In this paper, we explore the feasibility of leveraging language as a naturally high-quality supervision for chest CT imaging. In light of the limited availability of image-report pairs, we bootstrap the understanding of 3D chest CT images by distilling chest-related diagnostic knowledge from an extensively pre-trained 2D X-ray expert model. Specifically, we propose a language-guided retrieval method to match each 3D CT image with its semantically closest 2D X-ray image, and perform pair-wise and semantic relation knowledge distillation. Subsequently, we use contrastive learning to align images and reports within the same patient while distinguishing them from the other patients. However, the challenge arises when patients have similar semantic diagnoses, such as healthy patients, potentially confusing if treated as negatives. We introduce a robust contrastive learning that identifies and corrects these false negatives. We train our model with over 12,000 pairs of chest CT images and radiology reports. Extensive experiments across multiple scenarios, including zero-shot learning, report generation, and fine-tuning processes, demonstrate the model's feasibility in interpreting chest CT images.