# Large language models provide unsafe answers to patient-posed medical questions

arXiv: https://arxiv.org/abs/2507.18905 · HTML: https://mlsanity.com/paper/2507.18905/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Builds on (most cited first, 14 of 14)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- [MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models](https://mlsanity.com/paper/2403.03744/) (2024)
- [MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2404.05590/) (2024)
- [Large Language Models are overconfident and amplify human bias](https://mlsanity.com/paper/2505.02151/) (2025)
- [Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning](https://mlsanity.com/paper/2402.11537/) (2024)
- [Misspellings in Natural Language Processing: A survey](https://mlsanity.com/paper/2501.16836/) (2025)

## Built on by (most cited first, 18 of 18)

- [Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Setting](https://mlsanity.com/paper/2512.10780/) (2025)
- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [Quantifying and Mitigating Premature Closure in Frontier LLMs](https://mlsanity.com/paper/2605.15000/) (2026)
- [MIRA: A Bilingual Benchmark for Medical Information Response Audit](https://mlsanity.com/paper/2605.28025/) (2026)
- [MultiTurnPSB: Evaluating Multi-Turn Jailbreak Attacks an dClassifier-Based Defenses for Medical AI Safety](https://mlsanity.com/paper/2606.02630/) (2026)
- [Compartmentalised Agentic Reasoning for Clinical NLI](https://mlsanity.com/paper/2509.10222/) (2025)
- [Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditions](https://mlsanity.com/paper/2601.09721/) (2025)
- [GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration](https://mlsanity.com/paper/2605.24636/) (2026)
- [TriEval: A Resource-Efficient Pipeline for LLM Bias, Toxicity, and Truthfulness Assessment](https://mlsanity.com/paper/2606.03036/) (2026)
- [Agentic systems for breast cancer treatment recommendations](https://mlsanity.com/paper/2607.12051/) (2026)
- [MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection](https://mlsanity.com/paper/2607.15166/) (2026)
- [The Mirage of LLM Guardrails: A Case Study in AI-Assisted Medical Note Manipulation](https://mlsanity.com/paper/2607.24859/) (2026)
- [A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queries](https://mlsanity.com/paper/2608.19875/) (2026)
- [TSWAP: A Multilingual Retrieval-Augmented Thai Wellness Advisor](https://mlsanity.com/paper/2608.22917/) (2026)
- [SingProbe Technical Report](https://mlsanity.com/paper/2608.30703/) (2026)
- [Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection](https://mlsanity.com/paper/2609.03953/) (2026)
- [Challenges of Auditing: Variability in Outputs of Large Language Models for Health](https://mlsanity.com/paper/2609.16590/) (2026)
- [Large Language Models Versus Physicians in Traditional Chinese Medicine: A Real-World Clinical Case Evaluation](https://mlsanity.com/paper/2609.17544/) (2026)

## Abstract (arXiv)

Millions of patients are already using large language model (LLM) chatbots for medical advice on a regular basis, raising patient safety concerns. This physician-led red-teaming study compares the safety of four publicly available chatbots--Claude by Anthropic, Gemini by Google, GPT-4o by OpenAI, and Llama3-70B by Meta--on a new dataset, HealthAdvice, using an evaluation framework that enables quantitative and qualitative analysis. In total, 888 chatbot responses are evaluated for 222 patient-posed advice-seeking medical questions on primary care topics spanning internal medicine, women's health, and pediatrics. We find statistically significant differences between chatbots. The rate of problematic responses varies from 21.6 percent (Claude) to 43.2 percent (Llama), with unsafe responses varying from 5 percent (Claude) to 13 percent (GPT-4o, Llama). Qualitative results reveal chatbot responses with the potential to lead to serious patient harm. This study suggests that millions of patients could be receiving unsafe medical advice from publicly available chatbots, and further work is needed to improve the clinical safety of these powerful tools.