mlsanityData to 7 Oct 2026

MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment

arXiv 2403.10635 ↗First posted 15 Mar 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Pathology and anatomy are two essential groups of semantics in medical data. Pathology describes what the diseases are, while anatomy explains where the diseases occur. They describe diseases from different perspectives, providing complementary insights into diseases. Thus, properly understanding these semantics and their relationships can enhance medical vision-language models (VLMs). However, pathology and anatomy semantics are usually entangled in medical data, hindering VLMs from explicitly modeling these semantics and their relationships. To address this challenge, we propose MeDSLIP, a novel Medical Dual-Stream Language-Image Pre-training pipeline, to disentangle pathology and anatomy semantics and model the relationships between them. We introduce a dual-stream mechanism in MeDSLIP to explicitly disentangle medical semantics into pathology-relevant and anatomy-relevant streams and align visual and textual information within each stream. Furthermore, we propose an interaction modeling module with prototypical contrastive learning loss and intra-image contrastive learning loss to regularize the relationships between pathology and anatomy semantics. We apply MeDSLIP to chest X-ray analysis and conduct comprehensive evaluations with four benchmark datasets: NIH CXR14, RSNA Pneumonia, SIIM-ACR Pneumothorax, and COVIDx CXR-4. The results demonstrate MeDSLIP's superior generalizability and transferability across different scenarios. The code is available at https://github.com/Shef-AIRE/MeDSLIP, and the pre-trained model is released at https://huggingface.co/pykale/MeDSLIP.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

25 of 36 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Publicly Available Clinical BERT Embeddingscited by 995
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  5. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  7. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  8. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  9. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  10. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  11. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  12. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
13 more
  1. 2019Deep learning for cardiac image segmentation: A reviewcited by 131
  2. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  4. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 81
  5. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  6. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  7. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  8. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimizationcited by 43
  9. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  10. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  11. 2023UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classificationcited by 21
  12. 2024Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Promptingcited by 10
  13. 2023K-Diag: Knowledge-enhanced Disease Diagnosis in Radiographic Imagingcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray
  2. 2025MedFILIP: Medical Fine-grained Language-Image Pre-training
  3. 2025Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation
  4. 2026MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models
  5. 2025Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models
  6. 2025VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
  7. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training
  8. 2025Distribution-Based Masked Medical Vision-Language Model Using Structured Reports