mlsanityData to 7 Oct 2026

AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles

arXiv 2412.16701 ↗First posted 21 Dec 2024cs.IR cs.CLcited by 3

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Recent advancements in generative AI have fostered the development of highly adept Large Language Models (LLMs) that integrate diverse data types to empower decision-making. Among these, multimodal retrieval-augmented generation (RAG) applications are promising because they combine the strengths of information retrieval and generative models, enhancing their utility across various domains, including clinical use cases. This paper introduces AlzheimerRAG, a Multimodal RAG application for clinical use cases, primarily focusing on Alzheimer's Disease case studies from PubMed articles. This application incorporates cross-modal attention fusion techniques to integrate textual and visual data processing by efficiently indexing and accessing vast amounts of biomedical literature. Our experimental results, compared to benchmarks such as BioASQ and PubMedQA, have yielded improved performance in the retrieval and synthesis of domain-specific information. We also present a case study using our multimodal RAG in various Alzheimer's clinical scenarios. We infer that AlzheimerRAG can generate responses with accuracy non-inferior to humans and with low rates of hallucination.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 81 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

35 of 81 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models16 Oct
  2. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoning2 Aug
  3. 2024Benchmarking Retrieval-Augmented Generation for Medicine20 Feb
  4. 2024Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report29 Jan

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
19 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  6. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  7. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  8. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  9. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  10. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  11. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  12. 2023Cross-Modal Translation and Alignment for Survival Analysiscited by 70
  13. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  14. 2020Context-Aware Answer Extraction in Question Answeringcited by 13
  15. 2023Integrating UMLS Knowledge into Large Language Models for Medical Question Answeringcited by 10
  16. 2023Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialoguescited by 5
  17. 2024GFE-Mamba: Mamba-based AD Multi-modal Progression Assessment via Generative Feature Extraction from MCIcited by 4
  18. 2024PaCKD: Pattern-Clustered Knowledge Distillation for Compressing Memory Access Prediction Modelscited by 2
  19. 2024TMU at TREC Clinical Trials Track 2023cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2026Cognitive Prosthetic: An AI-Enabled Multimodal System for Episodic Recall in Knowledge Workcited by 1
  2. 2025Addressing accuracy and hallucination of LLMs in Alzheimer's disease research through knowledge graphscited by 0
  3. 2025Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generation
  2. 2025MIRA: A Novel Framework for Fusing Modalities in Medical RAG
  3. 2025BRAINS: A Retrieval-Augmented System for Alzheimer's Detection and Monitoring
  4. 2024BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine
  5. 2026Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
  6. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance
  7. 2026Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
  8. 2026MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine