mlsanityData to 7 Oct 2026

Capabilities of Claude Fable 5 on Biomedical Challenge Problems

arXiv 2607.10849 ↗First posted 12 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Frontier language models are increasingly evaluated on biomedical benchmarks, but two problems undermine most published evaluations: legacy benchmarks are near-saturated, and open-ended responses are graded by other language models. We evaluate Claude Fable 5, Anthropic's most capable publicly available model, across eight biomedical benchmarks, four text and four multimodal, using deterministic scoring against fixed answer keys throughout. We include two Claude predecessors and GPT-5 as baselines. Refusal is tracked as a distinct outcome in every result table. That decision produces the paper's central finding. Fable 5 refuses between 8.0% and 99.4% of questions depending on the benchmark, a pattern absent in both predecessors and in GPT-5. Once refused items are excluded from the denominator, Fable 5's accuracy exceeds or meets every other model on every benchmark in this study. We identify two distinguishable refusal patterns: one concentrating in basic-science and mechanism content across MedQA and MedXpertQA MM, confirmed independently on two benchmarks using each benchmark's own category labels; and a separate disease-domain pattern on RareBench, where inborn metabolic disease presentations are refused near-universally while adult-onset autoimmune presentations are not. The primary constraint on Fable 5's biomedical usefulness is willingness to engage, not capability once it does.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 17 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

10 of 17 references.

References · 10

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  6. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  7. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  8. 2024RareBench: Can LLMs Serve as Rare Diseases Specialists?cited by 45
  9. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  10. 2026MedCalc-Bench Doesn't Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluationcited by 3

Built on by

Not cited yet.