mlsanityData to 7 Oct 2026

Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images

arXiv 2302.14042 ↗First posted 27 Feb 2023cs.CVcited by 128

Not on the map. Why?

Cited by 51 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose a novel approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and radiology reports. We evaluate KAD on {four} external X-ray datasets and demonstrate that its zero-shot performance is not only comparable to that of fully-supervised models, but also superior to the average of three expert radiologists for three (out of five) pathologies with statistical significance. Moreover, when few-shot annotation is available, KAD outperforms all existing approaches in fine-tuning settings, demonstrating its potential for application in different clinical scenarios.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 46 of its 76 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

46 of 76 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
34 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  4. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
  5. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  6. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  7. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  8. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  9. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  10. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  11. 2019Entity, Relation, and Event Extraction with Contextualized Span Representationscited by 261
  12. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  13. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  14. 20203D Self-Supervised Methods for Medical Imagingcited by 134
  15. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  16. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  17. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  18. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  19. 2019Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cubecited by 76
  20. 2019Attention Based Glaucoma Detection: A Large-scale Database and CNN Modelcited by 75
  21. 2020Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representationscited by 69
  22. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  23. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Modelscited by 39
  24. 2020ChestX-Det10: Chest X-ray Dataset on Detection of Thoracic Abnormalitiescited by 37
  25. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  26. 2019CANet: Cross-disease Attention Network for Joint Diabetic Retinopathy and Diabetic Macular Edema Gradingcited by 35
  27. 2019Embedding Human Knowledge into Deep Neural Network via Attention Mapcited by 33
  28. 2020Models Genesiscited by 26
  29. 2020CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Outputcited by 23
  30. 2021Contrastive Language-Image Pre-training for the Italian Languagecited by 19
  31. 2017Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learningcited by 14
  32. 2018Training Medical Image Analysis Systems like Radiologistscited by 13
  33. 2023K-Diag: Knowledge-enhanced Disease Diagnosis in Radiographic Imagingcited by 6
  34. 2021CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settingscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 33; 2025: 50; 2026: 24.

128 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  4. 2023CLIP in Medical Imaging: A Surveycited by 40
  5. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  6. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
  7. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  8. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  9. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  10. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  11. 2023Medical Vision Language Pretraining: A surveycited by 24
  12. 2024Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imagingcited by 24
38 more
  1. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
  2. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  3. 2023Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challengecited by 22
  4. 2023UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classificationcited by 21
  5. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  6. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  7. 2024Anatomical Structure-Guided Medical Vision-Language Pre-trainingcited by 18
  8. 2024Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learningcited by 18
  9. 2023Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentationcited by 17
  10. 2023Large-scale Long-tailed Disease Diagnosis on Radiology Imagescited by 17
  11. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  12. 2025PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusioncited by 17
  13. 2023Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretationcited by 14
  14. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  15. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14
  16. 2024A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysiscited by 14
  17. 2024Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generationcited by 14
  18. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-trainingcited by 14
  19. 2024Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applicationscited by 13
  20. 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 13
  21. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-trainingcited by 12
  22. 2024DeViDe: Faceted medical knowledge for improved medical vision-language pre-trainingcited by 10
  23. 2024Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generationcited by 10
  24. 2024UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modelingcited by 10
  25. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosiscited by 9
  26. 2025Uncertainty-Aware Large Language Models for Explainable Disease Diagnosiscited by 9
  27. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  28. 2025PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generationcited by 9
  29. 2024Large Language Model-informed ECG Dual Attention Network for Heart Failure Risk Predictioncited by 8
  30. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  31. 2024MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Modelcited by 7
  32. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 7
  33. 2023Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Modelscited by 6
  34. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rayscited by 6
  35. 2025SuPreME: A Supervised Pre-training Framework for Multimodal ECG Representation Learningcited by 6
  36. 2025LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?cited by 6
  37. 2025More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM eracited by 6
  38. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinementcited by 5

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models
  2. 2025X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoning
  3. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime
  4. 2026RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
  5. 2024Visual Prompt Engineering for Vision Language Models in Radiology
  6. 2025Benchmarking Chest X-ray Diagnosis Models Across Multinational Datasets
  7. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiology
  8. 2025Evaluating Vision Language Models (VLMs) for Radiology: A Comprehensive Analysis