mlsanityData to 14 Oct 2026

Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework

arXiv 2403.07636 ↗First posted 12 Mar 2024cs.CVcited by 27

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This leads to the misalignment with the target disease's textual representation. In this paper, we introduce a novel VLP framework designed to dissect disease descriptions into their fundamental aspects, leveraging prior knowledge about the visual manifestations of pathologies. This is achieved by consulting a large language model and medical experts. Integrating a Transformer module, our approach aligns an input image with the diverse elements of a disease, generating aspect-centric image representations. By consolidating the matches from each aspect, we improve the compatibility between an image and its associated disease. Additionally, capitalizing on the aspect-oriented representations, we present a dual-head Transformer tailored to process known and unknown diseases, optimizing the comprehensive detection efficacy. Conducting experiments on seven downstream datasets, ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively. Our code is released at https://github.com/HieuPhan33/MAVL.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 51 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

40 of 51 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  12. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
28 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  6. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  7. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  8. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  9. 2022Visual Classification via Description from Large Language Modelscited by 336
  10. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  11. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
  12. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  13. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 309
  14. 2022Post-hoc Concept Bottleneck Modelscited by 293
  15. 2023Label-Free Concept Bottleneck Modelscited by 285
  16. 2019ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed datacited by 261
  17. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  18. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  19. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  20. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  21. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  22. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  23. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  24. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  25. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  26. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  27. 2021COVID-Net CXR-2: An Enhanced Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-ray Imagescited by 22
  28. 2022Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rayscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 8.

27 papers.

Sort:

By area:

  1. 2025Interactive Medical Image Analysis with Concept-based Similarity Reasoningcited by 11
  2. 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?cited by 7
  3. 2024Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignmentcited by 7
  4. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 7
  5. 2025LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?cited by 6
  6. 2025Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputscited by 5
  7. 2025Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detectioncited by 3
  8. 2025Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Groundingcited by 3
  9. 2025XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiographycited by 3
  10. 2026Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learningcited by 3
  11. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?cited by 2
  12. 2025Boosting Medical Visual Understanding From Multi-Granular Language Learningcited by 2
15 more
  1. 2024MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignmentcited by 1
  2. 2025Anatomy-Aware Text-Visual Fusion with Dual-Perspective Prompts for Fine-Grained Lumbar Spine Segmentationcited by 1
  3. 2025MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysiscited by 1
  4. 2025PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathologycited by 1
  5. 2026KEPIL: Knowledge-Enhanced Prompt-Image Learning for Prompt-Robust Disease Detectioncited by 1
  6. 2025Progressive Local Alignment for Medical Multimodal Pre-trainingcited by 0
  7. 2025Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridgingcited by 0
  8. 2025Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learningcited by 0
  9. 2025Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resourcescited by 0
  10. 2026Representation Learning with Semantic-aware Instance and Sparse Token Alignmentscited by 0
  11. 2026HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Expertscited by 0
  12. 2026MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentationcited by 0
  13. 2026AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via α-Corrected Binary Cross Entropy and Factorized Latent Supervisioncited by 0
  14. 2026Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classificationcited by 0
  15. 2026UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions
  2. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding
  3. 2023Unified Medical Image Pre-training in Language-Guided Common Semantic Space
  4. 2026FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology
  5. 2024Disease-informed Adaptation of Vision-Language Models
  6. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  7. 2026PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology
  8. 2025Distribution-Based Masked Medical Vision-Language Model Using Structured Reports