mlsanityData to 7 Oct 2026

MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology

arXiv 2301.02228 ↗First posted 5 Jan 2023eess.IV cs.CL cs.CVcited by 149

Not on the map. Why?

Cited by 61 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice. In particular, we make the following contributions: First, unlike existing works that directly process the raw reports, we adopt a novel triplet extraction module to extract the medical-related information, avoiding unnecessary complexity from language grammar and enhancing the supervision signals; Second, we propose a novel triplet encoding module with entity translation by querying a knowledge base, to exploit the rich domain knowledge in medical field, and implicitly build relationships between medical entities in the language embedding space; Third, we propose to use a Transformer-based fusion model for spatially aligning the entity description with visual signals at the image patch level, enabling the ability for medical diagnosis; Fourth, we conduct thorough experiments to validate the effectiveness of our architecture, and benchmark on numerous public benchmarks, e.g., ChestX-ray14, RSNA Pneumonia, SIIM-ACR Pneumothorax, COVIDx CXR-2, COVID Rural, and EdemaSeverity. In both zero-shot and fine-tuning settings, our model has demonstrated strong performance compared with the former methods on disease classification and grounding.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 42 of its 74 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

42 of 74 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2019Publicly Available Clinical BERT Embeddingscited by 995
  11. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
30 more
  1. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  2. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  3. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  4. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  5. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  6. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  7. 2019ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed datacited by 260
  8. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  9. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  10. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  11. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  12. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 82
  13. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  14. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  15. 2019Attention Based Glaucoma Detection: A Large-scale Database and CNN Modelcited by 75
  16. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  17. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  18. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 42
  19. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  20. 2019CANet: Cross-disease Attention Network for Joint Diabetic Retinopathy and Diabetic Macular Edema Gradingcited by 35
  21. 2019Embedding Human Knowledge into Deep Neural Network via Attention Mapcited by 33
  22. 2019Cross-Modal Data Programming Enables Rapid Medical Machine Learningcited by 32
  23. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  24. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  25. 2020CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Outputcited by 23
  26. 2021COVID-Net CXR-2: An Enhanced Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-ray Imagescited by 22
  27. 2022Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rayscited by 21
  28. 2021Contrastive Language-Image Pre-training for the Italian Languagecited by 19
  29. 2017Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learningcited by 14
  30. 2018Training Medical Image Analysis Systems like Radiologistscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 48; 2025: 44; 2026: 32.

149 papers.

Sort:

By area:

  1. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  2. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  3. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  4. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  5. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  6. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  7. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  8. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  9. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  10. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  11. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
  12. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 35
38 more
  1. 2024Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammographycited by 35
  2. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  3. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  4. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  5. 2024RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reportscited by 29
  6. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Modelscited by 28
  7. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 28
  8. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  9. 2023Medical Vision Language Pretraining: A surveycited by 24
  10. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  11. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-trainingcited by 22
  12. 2024EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learningcited by 22
  13. 2024Few-shot Adaptation of Medical Vision-Language Modelscited by 22
  14. 2023UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classificationcited by 21
  15. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  16. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  17. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  18. 2023Spatially Covariant Image Registration with Text Promptscited by 18
  19. 2024Anatomical Structure-Guided Medical Vision-Language Pre-trainingcited by 18
  20. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  21. 2023Large-scale Long-tailed Disease Diagnosis on Radiology Imagescited by 17
  22. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  23. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
  24. 2024Eye-gaze Guided Multi-modal Alignment for Medical Representation Learningcited by 16
  25. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  26. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14
  27. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  28. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-trainingcited by 14
  29. 2023UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-trainingcited by 13
  30. 2024Uncertainty-aware Medical Diagnostic Phrase Identification and Groundingcited by 13
  31. 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 13
  32. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-trainingcited by 12
  33. 2023RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extractioncited by 11
  34. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  35. 2024DeViDe: Faceted medical knowledge for improved medical vision-language pre-trainingcited by 10
  36. 2025Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysiscited by 10
  37. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  38. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9

50 most cited of 149 All 149 (JSON)