mlsanityData to 14 Oct 2026

How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

arXiv 2610.18529 ↗First posted 13 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains under explored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev demonstrates strong robustness, achieving the lowest decision reversal rate and relatively small accuracy fluctuations. Among evaluated open-source alternatives, only StartLux exhibits comparable robustness. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 43 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

19 of 43 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  7. 2020Selective Question Answering under Domain Shiftcited by 238
  8. 2024MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notescited by 80
  9. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 70
  10. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 64
  11. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 62
  12. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 40
7 more
  1. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
  2. 2026Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factualitycited by 11
  3. 2026JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Modelscited by 10
  4. 2026Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibrationcited by 3
  5. 2025RxSafeBench: Identifying Medication Safety Issues of Large Language Models in Simulated Consultationcited by 2
  6. 2026Jev in Medicine: A Benchmark Evaluationcited by 2
  7. 2026CARE-Bench: Benchmarking Patient-Facing LLM Triagecited by 1

Built on by

Not cited yet.