mlsanityData to 7 Oct 2026

A collective capability boundary in frontier large language models on guideline-conformant and case-specific oncology decision-making

arXiv 2608.28592 ↗First posted 8 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) achieve high scores on medical knowledge examinations, yet real-world oncology is not a knowledge test--it is a sequence of guideline-pathway choices, escalation judgments, and commitments under uncertainty. Existing benchmarks largely measure factual recall, leaving open whether frontier LLMs share decision-path blind spots that combining models cannot fix. We built the Oncology Decision Boundary Benchmark (ODBB)--2,005 oncology decision points across NCCN guidelines and colorectal cancer cases--and evaluated nine frontier LLMs (four closed-source, five open-weight families) released between June 2025 and April 2026. A fully deterministic scorer (zero LLM inference) classified outputs into 14 failure types, independently validated by two oncologists (Cohen's weighted kappa = 0.939 and 0.790) on a 225-item stratified sample. Treating the nine as a pooled super-model, 42.1% (Wilson 95% CI 40.0--44.3%) of all items--35.7% of the 1,586 NCCN items and 66.4% of the 419 colorectal-cancer cases--were answered correctly by none, with failures concentrated in choosing between guideline pathways before reasoning within any: a consistent blind spot in clinical meta-judgment that likely requires architectural intervention rather than more training data. Two models tuned for decisiveness (GPT-5.5, Gemini 3.1 Pro Preview) made unsafe commitments three to five times more often than the seven cautious models without scoring higher. In 3--9% of items, models stated the correct next clinical step yet did not commit to it--failures of decision, not knowledge. Model quality is no longer the primary bottleneck for clinical LLM deployment; the binding constraint is the assumption that any single model can be the sole basis for a clinical decision. Progress requires architectures that detect when a model reaches its competence boundary and route the decision to a clinician.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 68 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

29 of 68 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  12. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
17 more
  1. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  2. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  3. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  4. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  5. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  6. 2023How is ChatGPT's behavior changing over time?cited by 265
  7. 2024Capabilities of Gemini Models in Medicinecited by 231
  8. 2022Can large language models reason about medical questions?cited by 194
  9. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  10. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  11. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  12. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  13. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  14. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  15. 2023(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIscited by 21
  16. 2024CancerLLM: A Large Language Model in Cancer Domaincited by 11
  17. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelinescited by 7

Built on by

Not cited yet.