# Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework

arXiv: https://arxiv.org/abs/2403.07636 · HTML: https://mlsanity.com/paper/2403.07636/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 40 of its 51 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)

## Built on by (most cited first, 27 of 27)

- [Interactive Medical Image Analysis with Concept-based Similarity Reasoning](https://mlsanity.com/paper/2503.06873/) (2025)
- [Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?](https://mlsanity.com/paper/2410.13523/) (2024)
- [Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment](https://mlsanity.com/paper/2410.15744/) (2024)
- [KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoning](https://mlsanity.com/paper/2509.24816/) (2025)
- [LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?](https://mlsanity.com/paper/2503.07487/) (2025)
- [Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs](https://mlsanity.com/paper/2502.17900/) (2025)
- [Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection](https://mlsanity.com/paper/2502.16223/) (2025)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography](https://mlsanity.com/paper/2510.19599/) (2025)
- [Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning](https://mlsanity.com/paper/2604.13598/) (2026)
- [How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?](https://mlsanity.com/paper/2409.00543/) (2024)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment](https://mlsanity.com/paper/2403.10635/) (2024)
- [Anatomy-Aware Text-Visual Fusion with Dual-Perspective Prompts for Fine-Grained Lumbar Spine Segmentation](https://mlsanity.com/paper/2504.03476/) (2025)
- [MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2508.03441/) (2025)
- [PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology](https://mlsanity.com/paper/2509.06105/) (2025)
- [KEPIL: Knowledge-Enhanced Prompt-Image Learning for Prompt-Robust Disease Detection](https://mlsanity.com/paper/2605.09132/) (2026)
- [Progressive Local Alignment for Medical Multimodal Pre-training](https://mlsanity.com/paper/2502.18047/) (2025)
- [Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging](https://mlsanity.com/paper/2505.21698/) (2025)
- [Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning](https://mlsanity.com/paper/2511.18104/) (2025)
- [Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources](https://mlsanity.com/paper/2512.02438/) (2025)
- [Representation Learning with Semantic-aware Instance and Sparse Token Alignments](https://mlsanity.com/paper/2601.08165/) (2026)
- [HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts](https://mlsanity.com/paper/2608.02252/) (2026)
- [MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation](https://mlsanity.com/paper/2608.13690/) (2026)
- [AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via α-Corrected Binary Cross Entropy and Factorized Latent Supervision](https://mlsanity.com/paper/2609.01757/) (2026)
- [Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classification](https://mlsanity.com/paper/2609.13303/) (2026)
- [UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learning](https://mlsanity.com/paper/2609.31298/) (2026)

## Abstract (arXiv)

Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This leads to the misalignment with the target disease's textual representation. In this paper, we introduce a novel VLP framework designed to dissect disease descriptions into their fundamental aspects, leveraging prior knowledge about the visual manifestations of pathologies. This is achieved by consulting a large language model and medical experts. Integrating a Transformer module, our approach aligns an input image with the diverse elements of a disease, generating aspect-centric image representations. By consolidating the matches from each aspect, we improve the compatibility between an image and its associated disease. Additionally, capitalizing on the aspect-oriented representations, we present a dual-head Transformer tailored to process known and unknown diseases, optimizing the comprehensive detection efficacy. Conducting experiments on seven downstream datasets, ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively. Our code is released at https://github.com/HieuPhan33/MAVL.