CRS-Bench: A Reference-Relative Reliability Benchmark for Medical Image Encoders
Not on the map. Why?
Abstract (arXiv)
Pretrained image encoders are central to medical image classification, where expert annotation is costly and task-specific cohorts are often limited. As the model space expands from general-purpose to broad-medical and specialty-specific encoders, selecting the representation becomes a substantive modeling decision. Clean-test discrimination alone is insufficient for this purpose: encoders with similar AUROC can differ in calibration, label efficiency, and stability under acquisition perturbations or distribution shift. We introduce CRS-Bench, a controlled benchmark for multi-objective medical encoder selection. CRS-Bench evaluates 15 pretrained encoder families across dermatology, ophthalmology, and radiology using ISIC 2019, APTOS 2019, and CheXpert, with CheXpert-to-MIMIC-CXR as an observed institutional shift, yielding 17,575 controlled run records and 3,515 seed-aggregated metric rows. Each encoder is characterized along four operational reliability dimensions: discrimination, calibration, label efficiency, and robustness. We summarize these dimensions using the Clinical Reliability Score (CRS), a Pareto-aware, reference-relative score combining dominance, profile balance, and worst-axis performance. AUROC and CRS are positively associated but not decision-equivalent: 21 of 105 pairwise orderings reverse, with a mean absolute rank displacement of 1.87. Paired-seed bootstrap analysis identifies PanDerm, MedSigLIP, and MedGemma as a stable leading reliability tier rather than a statistically resolved single leader. CRS-Bench provides a controlled framework for selecting medical image encoders from multi-axis reliability profiles rather than clean-test AUROC alone.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 22 Aug 2026.
Builds on
23 of 34 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 669
11 more
- 2025MedGemma Technical Reportcited by 436
- 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
- 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
- 2019BCN20000: Dermoscopic Lesions in the Wildcited by 225
- 2019Causality matters in medical imagingcited by 193
- 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 163
- 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
- 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
- 2024A Multimodal Vision Foundation Model for Clinical Dermatologycited by 50
- 2024FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Modelscited by 47
- 2024A Large-scale Medical Visual Task Adaptation Benchmarkcited by 3
Built on by
Not cited yet.