mlsanityData to 14 Oct 2026

TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent

arXiv 2608.10258 ↗First posted 10 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) increasingly provide conversational health information that may influence treatment decisions, yet existing benchmarks do not isolate whether medication-safety boundaries persist across follow-ups after explicit self-treatment intent. We introduce TAF-MED, a physician-reviewed benchmark of 500 fixed three-turn scenarios, and evaluate eight LLMs across 4,000 conversations. A rubric-based automated judge labelled responses as SAFE, LEAKY, or UNSAFE, and two physicians independently annotated a model-balanced random subset of 400 conversations. We assessed unsafe guidance, collapse after a strictly SAFE initial response, and model-ranking stability. Overall, 71.6% of conversations contained an UNSAFE response, and 61.4% of those beginning with a strictly SAFE response later collapsed to UNSAFE; model-level collapse rates ranged from 24.4% to 96.2%. Four of 28 model pairs reversed order between initial unsafe and collapse rates. Automated labels achieved 94.3% agreement with the adjudicated physician reference (kappa = 0.895). These findings show that first-turn safety is an incomplete proxy for conversational safety persistence and motivate evaluation across complete dialogue trajectories. We will release TAF-MED on Hugging Face to support reproducible research on multi-turn medical safety.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 23 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

7 of 23 references.

References · 7

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  4. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  5. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safetycited by 8
  6. 2025"What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasetscited by 5
  7. 2026MultiTurnPSB: Evaluating Multi-Turn Jailbreak Attacks an dClassifier-Based Defenses for Medical AI Safetycited by 2

Built on by

Not cited yet.