mlsanityData to 7 Oct 2026

Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation

arXiv 2602.15650 ↗First posted 17 Feb 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Radiology Report Generation (RRG) through Vision-Language Models (VLMs) promises to reduce documentation burden, improve reporting consistency, and accelerate clinical workflows. However, their clinical adoption remains limited by the lack of interpretability and the tendency to hallucinate findings misaligned with imaging evidence. Existing research typically treats interpretability and accuracy as separate objectives, with concept-based explainability techniques focusing primarily on transparency, while Retrieval-Augmented Generation (RAG) methods targeting factual grounding through external retrieval. We present Concept-Enhanced Multimodal RAG (CEMRAG), a unified framework that decomposes visual representations into interpretable clinical concepts and integrates them with multimodal RAG. This approach exploits enriched contextual prompts for RRG, improving both interpretability and factual accuracy. Experiments on MIMIC-CXR and IU X-Ray across multiple VLM architectures, training regimes, and retrieval configurations demonstrate consistent improvements over both conventional RAG and concept-only baselines on clinical accuracy metrics and standard NLP measures. These results challenge the assumed trade-off between interpretability and performance, showing that transparent visual concepts can enhance rather than compromise diagnostic accuracy in medical VLMs. Our modular design decomposes interpretability into visual transparency and structured language model conditioning, providing a principled pathway toward clinically trustworthy AI-assisted radiology.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 60 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

48 of 60 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025MIRA: A Novel Framework for Fusing Modalities in Medical RAG10 Jul
  2. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models16 Oct
  3. 2024OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography30 Aug
  4. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models6 Jul
  5. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar
  6. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study21 Feb
  7. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training20 Oct
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  8. 2020Concept Bottleneck Modelscited by 1,022
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  12. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
28 more
  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024The Faiss librarycited by 370
  3. 2023Explainability for Large Language Models: A Surveycited by 331
  4. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  5. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  6. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  7. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  8. 2025Open Problems in Mechanistic Interpretabilitycited by 173
  9. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  10. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  11. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  12. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  13. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  14. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  15. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  16. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 82
  17. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  18. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  19. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  20. 2024A Concept-Based Explainability Framework for Large Multimodal Modelscited by 39
  21. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  22. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  23. 2024Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIPcited by 19
  24. 2024LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Taskscited by 12
  25. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  26. 2025Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generationcited by 8
  27. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decodingcited by 6
  28. 2024Application of NotebookLM, a Large Language Model with Retrieval-Augmented Generation, for Lung Cancer Stagingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CBM-RAG: Demonstrating Enhanced Interpretability in Radiology Report Generation with Multi-Agent RAG and Concept Bottleneck Models
  2. 2023MAIRA-1: A specialised large multimodal model for radiology report generation
  3. 2026Improving Health Literacy through Lay Summarization of Radiological Reports: An Evaluation of BioNER and Retrieval-Augmented Generation
  4. 2025RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment
  5. 2024Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG
  6. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
  7. 2025RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extraction
  8. 2026Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context