mlsanityData to 7 Oct 2026

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

arXiv 2609.21763 ↗First posted 18 Sep 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

A medical model's benchmark score does not establish that the same conclusion holds under a different evaluation. This study tests whether claims about model ranking, score reliability and screening performance survive changes in cohort, prompt, negative spectrum, specified prevalence and operating threshold. We audit three medical vision-language models (BioMedCLIP, CheXficient, and MedSigLIP) and a general-domain OpenCLIP comparator on 12,200 chest radiograph records from four datasets (Montgomery, Shenzhen, TBX11K, and VinDr-CXR). Five fixed prompt families yield 244,000 model--image--prompt scores. No model leads every cohort and reliability criterion. Prompt-family changes alter AUROC in 21 of 48 multiplicity-controlled comparisons. Replacing healthy controls with sick non-tuberculosis controls reduces AUROC by 0.075--0.306 across all four models. On VinDr-CXR, the three medical models distinguish tuberculosis from no-finding controls substantially better than from pneumonia or lung tumor; their AUROC point estimates for both named diseases fall below 0.5. CheXficient has documented VinDr-CXR pretraining exposure, which limits the interpretation of its results. Thresholds chosen for 95% sensitivity on TBX11K training retain that constraint by point estimate in only four of sixteen target evaluations. A five-seed supervised source model reaches 0.999 AUROC on TBX11K validation but 0.629 on each of two external cohorts. Conservative exclusion of perceptual-overlap candidates narrows this gap without closing it. These retrospective, single-task results show that discrimination, score reliability and threshold retention support different portability claims. Evidence for chest X-ray tuberculosis screening should identify the complete evaluation specification rather than attribute clinical portability to a checkpoint alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 31 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

16 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models23 Sep
  2. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?31 Aug

Other references · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  5. 2017Selective Classification for Deep Neural Networkscited by 799
  6. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  7. 2025MedGemma Technical Reportcited by 398
  8. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  9. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  10. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  11. 2023FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcarecited by 59
  12. 2024FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Modelscited by 47
2 more
  1. 2025Robustness tests for biomedical foundation models should tailor to specificationscited by 3
  2. 2026A data- and compute-efficient chest X-ray foundation model beyond aggressive scalingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-language models for chest radiography do not always need the image
  2. 2026Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
  3. 2026Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models
  4. 2025Advancing Chronic Tuberculosis Diagnostics Using Vision-Language Models: A Multi modal Framework for Precision Analysis
  5. 2025Vision-Language Models for Acute Tuberculosis Diagnosis: A Multimodal Approach Combining Imaging and Clinical Data
  6. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
  7. 2026Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance
  8. 2026MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models