# Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions

arXiv: https://arxiv.org/abs/2608.13786 · HTML: https://mlsanity.com/paper/2608.13786/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 49 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models](https://mlsanity.com/paper/2311.10054/) (2023)
- [LitSearch: A Retrieval Benchmark for Scientific Literature Search](https://mlsanity.com/paper/2407.18940/) (2024)
- [Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias](https://mlsanity.com/paper/2405.15739/) (2024)
- [Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition](https://mlsanity.com/paper/2412.13612/) (2024)
- [MedCite: Can Language Models Generate Verifiable Text for Medicine?](https://mlsanity.com/paper/2506.06605/) (2025)
- [FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG](https://mlsanity.com/paper/2601.05866/) (2026)
- [PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading](https://mlsanity.com/paper/2510.22242/) (2025)
- [Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution](https://mlsanity.com/paper/2603.05308/) (2026)
- [Errors in AI-Assisted Retrieval of Medical Literature: A Comparative Study](https://mlsanity.com/paper/2603.22344/) (2026)
- [This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA](https://mlsanity.com/paper/2604.05051/) (2026)
- [DeepER-Med: Advancing Deep Evidence-Based Research in Medicine Through Agentic AI](https://mlsanity.com/paper/2604.15456/) (2026)

## Abstract (arXiv)

Large language model (LLM) chatbots are increasingly used to answer clinical questions with citations to relevant studies, yet the quality of retrieved evidence and factors influencing study selection remain unclear. We evaluated three general-purpose LLM chatbots (Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5) using 20 clinical questions adapted from 2026 Cochrane reviews. We simulated patient, clinician, and evidence-synthesis researcher roles and obtained four independent responses for each chatbot-role-question combination, yielding 720 responses (3 chatbots × 3 user roles × 4 repetitions × 20 review questions). Chatbots were asked to support their answers with primary clinical citations, which were benchmarked against the included and excluded study sets of the corresponding Cochrane reviews. On average, a single response retrieved 39.2% ± 29.8% of the corresponding Cochrane included-study set and 5.0% ± 9.4% of the excluded-study set. Recall of included studies varied significantly by model and user role. ChatGPT achieved higher recall than Claude or Gemini (63.1% ± 29.5% vs. 37.0% ± 23.8% vs. 17.3% ± 13.1%; blocked permutation test, p=2.0×10-5), and the researcher role yielded higher recall than the clinician or patient roles (42.8% ± 30.8% vs. 38.6% ± 28.9% vs. 36.1% ± 29.3%; p=2.0×10-5). Controlling for publication year, citations per year, and open-access status, sample size was the only significant predictor of retrieval: each doubling of sample size was associated with 50% higher odds of retrieval (odds ratio 1.50, 95% CI 1.24-1.81). These findings show that LLM chatbots can retrieve studies identified by expert reviewers, but retrieval varies substantially across models and user roles and favors larger clinical trials.