mlsanityData to 7 Oct 2026

Evaluating Search Engines and Large Language Models for Answering Health Questions

arXiv 2407.12468 ↗First posted 17 Jul 2024cs.IR cs.AIcited by 5

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Search engines (SEs) have traditionally been primary tools for information seeking, but the new Large Language Models (LLMs) are emerging as powerful alternatives, particularly for question-answering tasks. This study compares the performance of four popular SEs, seven LLMs, and retrieval-augmented (RAG) variants in answering 150 health-related questions from the TREC Health Misinformation (HM) Track. Results reveal SEs correctly answer between 50 and 70% of questions, often hindered by many retrieval results not responding to the health question. LLMs deliver higher accuracy, correctly answering about 80% of questions, though their performance is sensitive to input prompts. RAG methods significantly enhance smaller LLMs' effectiveness, improving accuracy by up to 30% by integrating retrieval evidence.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 24 of its 52 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

24 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Benchmarking Retrieval-Augmented Generation for Medicine20 Feb

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2019Passage Re-ranking with BERTcited by 629
11 more
  1. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  2. 2023Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineeringcited by 86
  3. 2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 74
  4. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  5. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  6. 2023Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformerscited by 22
  7. 2022No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrievalcited by 17
  8. 2023Search Still Matters: Information Retrieval in the Era of Generative AIcited by 17
  9. 2024Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarioscited by 13
  10. 2023Evaluation of AI Chatbots for Patient-Specific EHR Questionscited by 9
  11. 2024Dynamic Q&A of Clinical Documents with Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extractioncited by 6
  2. 2026EHRNavigator: A Multi-Agent System for Patient-Level Clinical Question Answering over Heterogeneous Electronic Health Recordscited by 1
  3. 2025Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answerscited by 0
  4. 2026Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answeringcited by 0
  5. 2026Challenges of Auditing: Variability in Outputs of Large Language Models for Healthcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Answering real-world clinical questions using large language model based systems
  2. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering
  3. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
  4. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology
  5. 2025Retrieval Augmented Generation Evaluation for Health Documents
  6. 2024The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation for Healthcare QA
  7. 2025Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases
  8. 2024Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report