mlsanityData to 7 Oct 2026

Benchmarking Retrieval-Augmented Generation for Medicine

arXiv 2402.13178 ↗First posted 20 Feb 2024cs.CL cs.AIcited by 326

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrieval-augmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MedRAG toolkit introduced in this work. Overall, MedRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4-level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the "lost-in-the-middle" effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine.

Key facts. Cited by 326 papers in the map (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 40 of its 67 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

40 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicine1 Mar

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
27 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  4. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  8. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  9. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  12. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  13. 2023Augmented Language Models: a Surveycited by 398
  14. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  15. 2022Can large language models reason about medical questions?cited by 194
  16. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  17. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  18. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  19. 2019A Question-Entailment Approach to Question Answeringcited by 133
  20. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  21. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  22. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  23. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  24. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  25. 2022BioLORD: Learning Ontological Representations from Definitions (for Biomedical Concepts and their Textual Descriptions)cited by 20
  26. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
  27. 2023Search Still Matters: Information Retrieval in the Era of Generative AIcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 77; 2025: 140; 2026: 109.

326 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  3. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  4. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  5. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  6. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  7. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  8. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 89
  11. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  12. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
38 more
  1. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  2. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  3. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  4. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  5. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  6. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  7. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 51
  8. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  9. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  10. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  11. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  12. 2025s3: You Don't Need That Much Data to Train a Search Agent via RLcited by 41
  13. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  14. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  15. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
  16. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  17. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  18. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  19. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  20. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 31
  21. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  22. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  23. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  24. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  25. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  26. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 28
  27. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
  28. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  29. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  30. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoorscited by 23
  31. 2025Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generationcited by 23
  32. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  33. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  34. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 22
  35. 2024RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Recordscited by 21
  36. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Datacited by 21
  37. 2024Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Modelscited by 21
  38. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20

50 most cited of 326 All 326 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering
  2. 2024Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report
  3. 2025Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerations
  4. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering
  5. 2026A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
  6. 2026Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
  7. 2025Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
  8. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines