mlsanityData to 7 Oct 2026

OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One

arXiv 2610.00381 ↗First posted 30 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Medical models are judged not only on correctness, but on whether reported confidence matches actual accuracy. Generalist multimodal medical models have expanded what a single model can perceive, yet they still express bounded decisions such as diagnoses, findings or cell counts as generated text, so the reported probability reflects the next token rather than the decision itself. Motivated by decision-native interfaces such as Jev, we introduce OmniMed-Jev, which represents each medical decision as a Choice, Noul or Score decision over a runtime-supplied candidate set and returns a full distribution over that set: mutually exclusive classes, binary presence of a finding, or a bounded ordered value. The design is omni in three respects: it accepts diverse imaging modalities, covers different prediction tasks, and expresses them through one candidate-conditioned probability model, so heterogeneous outputs become comparable probabilities rather than task-specific strings. In an interface-controlled comparison against a generative baseline trained on the same backbone, data and schedule, OmniMed-Jev's reported probabilities track observed correctness far more closely, reducing calibration error by up to an order of magnitude and reliability error by up to two, while point-prediction performance remains comparable; counting is the one family where the generative baseline stays ahead. Making the decision distribution the model's output is not a format change but what turns reported numbers into probabilities that mean what they say. These results support explicit decision modeling as a way to make reported confidence meaningful within the evaluated tasks, and they are not evidence of clinical readiness: the comparison cannot separate the interface from associated training differences, which we state alongside the results. Code is available at github.com/lytang63/OmniMed-Jev.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 67 of its 106 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

67 of 106 references.

References · 67

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  11. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
38 more
  1. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  2. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  3. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  4. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  5. 2024What Large Language Models Know and What People Think They Knowcited by 109
  6. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  7. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  8. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  9. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 79
  10. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  11. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 57
  12. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  13. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
  14. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  15. 2020ChestX-Det10: Chest X-ray Dataset on Detection of Thoracic Abnormalitiescited by 37
  16. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 37
  17. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  18. 2024Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Modelscited by 16
  19. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
  20. 2026Knowing When to Abstain: Medical LLMs Under Clinical Uncertaintycited by 13
  21. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 13
  22. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 13
  23. 2025Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?cited by 11
  24. 2025Cascaded Language Models for Cost-effective Human-AI Decision-Makingcited by 11
  25. 2026Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)cited by 11
  26. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  27. 2026Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agentscited by 9
  28. 2026REFLEX with Jev for Efficient Selective Control in LLM Agentscited by 9
  29. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 8
  30. 2026Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factualitycited by 8
  31. 2026Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Modelcited by 7
  32. 2025Uncertainty-aware abstention in medical diagnosis based on medical textscited by 6
  33. 2025Pitfalls of Conformal Predictions for Medical Image Classificationcited by 6
  34. 2026Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestrationcited by 6
  35. 2026Visual Jev: Accurate and Efficient Decisions from Shared Visual Contextcited by 6
  36. 2026Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigationcited by 5
  37. 2026Large Language Models Are Overconfident in Their Own Responsescited by 5
  38. 2026Fast Intent-Driven Service Orchestration with Jev for 6G Edge Networkscited by 5

50 most cited of 67 All 67 (JSON)

Built on by

Not cited yet.