mlsanityData to 14 Oct 2026

VLM-KG: Multimodal Radiology Knowledge Graph Generation

arXiv 2505.17042 ↗First posted 13 May 2025cs.CL cs.CV cs.IRcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated remarkable success in natural language generation, excelling at instruction following and structured output generation. Knowledge graphs play a crucial role in radiology, serving as valuable sources of factual information and enhancing various downstream tasks. However, generating radiology-specific knowledge graphs presents significant challenges due to the specialized language of radiology reports and the limited availability of domain-specific data. Existing solutions are predominantly unimodal, meaning they generate knowledge graphs only from radiology reports while excluding radiographic images. Additionally, they struggle with long-form radiology data due to limited context length. To address these limitations, we propose a novel multimodal VLM-based framework for knowledge graph generation in radiology. Our approach outperforms previous methods and introduces the first multimodal solution for radiology knowledge graph generation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 27 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

20 of 27 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  10. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
8 more
  1. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
  2. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  3. 2024An Introduction to Vision-Language Modelingcited by 178
  4. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  5. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  6. 2021VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labelscited by 27
  7. 2019Understanding Spatial Language in Radiology: Representation Framework, Annotation, and Spatial Relation Extraction from Chest X-ray Reports using Deep Learningcited by 12
  8. 2024LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologiescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models
  2. 2024SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models
  3. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  4. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
  5. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
  6. 2025RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment
  7. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
  8. 2026Efficient Multi-Granularity Knowledge Transfer for Radiology Report Generation