# Retrieval-augmented systems can be dangerous medical communicators

arXiv: https://arxiv.org/abs/2502.14898 · HTML: https://mlsanity.com/paper/2502.14898/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 15 of 15)

- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs](https://mlsanity.com/paper/2411.14199/) (2024)
- [When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour](https://mlsanity.com/paper/2311.09410/) (2023)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024)
- [Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses](https://mlsanity.com/paper/2410.22349/) (2024)
- [Argumentative Experience: Reducing Confirmation Bias on Controversial Issues through LLM-Generated Multi-Persona Debates](https://mlsanity.com/paper/2412.04629/) (2024)
- [Human-like Affective Cognition in Foundation Models](https://mlsanity.com/paper/2409.11733/) (2024)
- [The Usability of Pragmatic Communication in Regular Expression Synthesis](https://mlsanity.com/paper/2308.06656/) (2023)
- [WatChat: Explaining perplexing programs by debugging mental models](https://mlsanity.com/paper/2403.05334/) (2024)
- [A pragmatic account of the weak evidence effect](https://mlsanity.com/paper/2112.03799/) (2021)
- [Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models' Ability to Resist Requests for Misinformation](https://mlsanity.com/paper/2409.20385/) (2024)

## Built on by (most cited first, 5 of 5)

- [Forecasting Clinical Risk from Textual Time Series: Structuring Narratives for Temporal AI in Healthcare](https://mlsanity.com/paper/2504.10340/) (2025)
- [Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models](https://mlsanity.com/paper/2509.05691/) (2025)
- [Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.09349/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)
- [CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine](https://mlsanity.com/paper/2609.16301/) (2026)

## Abstract (arXiv)

Patients have long sought health information online, and increasingly, they are turning to generative AI to answer their health-related queries. Given the high stakes of the medical domain, techniques like retrieval-augmented generation and citation grounding have been widely promoted as methods to reduce hallucinations and improve the accuracy of AI-generated responses and have been widely adopted into search engines. This paper argues that even when these methods produce literally accurate content drawn from source documents sans hallucinations, they can still be highly misleading. Patients may derive significantly different interpretations from AI-generated outputs than they would from reading the original source material, let alone consulting a knowledgeable clinician. Through a large-scale query analysis on topics including disputed diagnoses and procedure safety, we support our argument with quantitative and qualitative evidence of the suboptimal answers resulting from current systems. In particular, we highlight how these models tend to decontextualize facts, omit critical relevant sources, and reinforce patient misconceptions or biases. We propose a series of recommendations -- such as the incorporation of communication pragmatics and enhanced comprehension of source documents -- that could help mitigate these issues and extend beyond the medical domain.