# Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions

arXiv: https://arxiv.org/abs/2408.00727 · HTML: https://mlsanity.com/paper/2408.00727/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 40 of its 45 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 30 of 40)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://mlsanity.com/paper/2305.15294/) (2023)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health](https://mlsanity.com/paper/2306.10070/) (2023)
- [BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text](https://mlsanity.com/paper/2403.18421/) (2024)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [Matching Patients to Clinical Trials with Large Language Models](https://mlsanity.com/paper/2307.15051/) (2023)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)

## Built on by (most cited first, 30 of 54)

- [From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents](https://mlsanity.com/paper/2506.18959/) (2025)
- [m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Models](https://mlsanity.com/paper/2504.00869/) (2025)
- [Rationale-Guided Retrieval Augmented Generation for Medical Question Answering](https://mlsanity.com/paper/2411.00300/) (2024)
- [RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models](https://mlsanity.com/paper/2412.02830/) (2024)
- [Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications](https://mlsanity.com/paper/2501.02460/) (2025)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [MedCite: Can Language Models Generate Verifiable Text for Medicine?](https://mlsanity.com/paper/2506.06605/) (2025)
- [From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations](https://mlsanity.com/paper/2509.16584/) (2025)
- [GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Development](https://mlsanity.com/paper/2503.00686/) (2025)
- [SearchRAG: Can Search Engines Be Helpful for LLM-based Medical Question Answering?](https://mlsanity.com/paper/2502.13233/) (2025)
- [Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models](https://mlsanity.com/paper/2505.14599/) (2025)
- [Multi-step retrieval and reasoning improves radiology question answering with large language models](https://mlsanity.com/paper/2508.00743/) (2025)
- [Med-R2: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine](https://mlsanity.com/paper/2501.11885/) (2025)
- [RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering](https://mlsanity.com/paper/2502.13361/) (2025)
- [DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients](https://mlsanity.com/paper/2505.19538/) (2025)
- [Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering](https://mlsanity.com/paper/2511.10900/) (2025)
- [HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings](https://mlsanity.com/paper/2410.13671/) (2024)
- [TARSE: Test-Time Adaptation via Retrieval of Skills and Experience for Reasoning Agents](https://mlsanity.com/paper/2603.01241/) (2026)
- [Entry-level guide to the use of large language models for medical research](https://mlsanity.com/paper/2410.18856/) (2024)
- [Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024](https://mlsanity.com/paper/2411.05762/) (2024)
- [Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering](https://mlsanity.com/paper/2509.00798/) (2025)
- [RAR2: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval](https://mlsanity.com/paper/2509.22713/) (2025)
- [CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction](https://mlsanity.com/paper/2604.16742/) (2026)
- [POLYRAG: Integrating Polyviews into Retrieval-Augmented Generation for Medical Applications](https://mlsanity.com/paper/2504.14917/) (2025)
- [CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering](https://mlsanity.com/paper/2508.18430/) (2025)
- [Build AI Assistants using Large Language Models and Agents to Enhance the Engineering Education of Biomechanics](https://mlsanity.com/paper/2511.15752/) (2025)
- [A Medical Multimodal Diagnostic Framework Integrating Vision-Language Models and Logic Tree Reasoning](https://mlsanity.com/paper/2512.21583/) (2025)
- [When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering](https://mlsanity.com/paper/2601.19827/) (2026)
- [MED-COPILOT: A Medical Assistant Powered by GraphRAG and Similar Patient Case Retrieval](https://mlsanity.com/paper/2603.00460/) (2026)
- [Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why](https://mlsanity.com/paper/2606.19602/) (2026)

## Abstract (arXiv)

The emergent abilities of large language models (LLMs) have demonstrated great potential in solving medical questions. They can possess considerable medical knowledge, but may still hallucinate and are inflexible in the knowledge updates. While Retrieval-Augmented Generation (RAG) has been proposed to enhance the medical question-answering capabilities of LLMs with external knowledge bases, it may still fail in complex cases where multiple rounds of information-seeking are required. To address such an issue, we propose iterative RAG for medicine (i-MedRAG), where LLMs can iteratively ask follow-up queries based on previous information-seeking attempts. In each iteration of i-MedRAG, the follow-up queries will be answered by a conventional RAG system and they will be further used to guide the query generation in the next iteration. Our experiments show the improved performance of various LLMs brought by i-MedRAG compared with conventional RAG on complex questions from clinical vignettes in the United States Medical Licensing Examination (USMLE), as well as various knowledge tests in the Massive Multitask Language Understanding (MMLU) dataset. Notably, our zero-shot i-MedRAG outperforms all existing prompt engineering and fine-tuning methods on GPT-3.5, achieving an accuracy of 69.68% on the MedQA dataset. In addition, we characterize the scaling properties of i-MedRAG with different iterations of follow-up queries and different numbers of queries per iteration. Our case studies show that i-MedRAG can flexibly ask follow-up queries to form reasoning chains, providing an in-depth analysis of medical questions. To the best of our knowledge, this is the first-of-its-kind study on incorporating follow-up queries into medical RAG. The implementation of i-MedRAG is available at https://github.com/Teddy-XiongGZ/MedRAG.