RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training

arXiv 2403.09948 ↗First posted 15 Mar 2024cs.CV cs.AIcited by 8

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The integration of artificial intelligence (AI) with radiology marks a transformative era in medicine. Vision foundation models have been adopted to enhance radiologic imaging analysis. However, the distinct complexities of radiologic 2D and 3D radiologic data pose unique challenges that existing models, pre-trained on general non-medical images, fail to address adequately. To bridge this gap and capitalize on the diagnostic precision required in radiologic imaging, we introduce Radiologic Contrastive Language-Image Pre-training (RadCLIP): a cross-modal vision-language foundational model that harnesses Vision Language Pre-training (VLP) framework to improve radiologic image analysis. Building upon Contrastive Language-Image Pre-training (CLIP), RadCLIP incorporates a slice pooling mechanism tailored for volumetric image analysis and is pre-trained using a large and diverse dataset of radiologic image-text pairs. The RadCLIP was pre-trained to effectively align radiologic images with their corresponding text annotations, creating a robust vision backbone for radiologic images. Extensive experiments demonstrate RadCLIP's superior performance in both uni-modal radiologic image classification and cross-modal image-text matching, highlighting its significant promise for improving diagnostic accuracy and efficiency in clinical settings. Our Key contributions include curating a large dataset with diverse radiologic 2D/3D radiologic image-text pairs, a slice pooling adapter using an attention mechanism for integrating 2D images, and comprehensive evaluations of RadCLIP on various radiologic downstream tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 89 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

44 of 89 references.

Closest ancestors · 2

Same direction, newest first

Other references · 42

2017Attention Is All You Needcited by 59,932
2015Deep Residual Learning for Image Recognitioncited by 51,809
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845

30 more

2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
2016Understanding intermediate layers using linear classifier probescited by 1,279
2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
2021Grounded Language-Image Pre-trainingcited by 1,147
2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
2017Deep Learning for Medical Image Analysiscited by 399
2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
2020MosMedData: Chest CT Scans With COVID-19 Related Findings Datasetcited by 103
2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
2022What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factorscited by 61
2022Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Searchcited by 38
2021COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer modelscited by 27
2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
2023CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Predictioncited by 16
2023Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentationcited by 11
2022Inflating 2D Convolution Weights for Efficient Generation of 3D Medical Imagescited by 8
2022Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?cited by 8
2021Learning Aligned Cross-Modal Representation for Generalized Zero-Shot Classificationcited by 4
2023Duke Spleen Data Set: A Publicly Available Spleen MRI and CT dataset for Training Segmentationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

Similar, not cited

Close in wording, no citation either way