mlsanityData to 14 Oct 2026

PROTON: Prototype-Based Test-Time Online OOD Detection for Medical VLMs

arXiv 2606.20913 ↗First posted 18 Jun 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Medical vision-language models (VLMs) enable zero-shot clinical image classification, yet reliably detecting out-of-distribution (OOD) inputs at deployment remains an open problem. No static scoring method works across all shift types: Maximum Concept Matching (MCM) on FLAIR achieves 76.4% AUROC for far-OOD but only 42.4% for covariate shifts such as ultra-wide-field fundus images, effectively random. We trace this to a structural mismatch: covariate-shifted inputs are indistinguishable from in-distribution samples in softmax space, yet occupy distinct regions in the VLM embedding space. To exploit this untapped signal, we propose PROTON (PROtotype-based Test-time ONline OOD detection), a lightweight post-hoc module that maintains an online prototype bank from high-confidence test predictions and adaptively fuses prototype distance with MCM scoring via stream-level variance statistics, requiring no model modification, training data, or prompt engineering. On the ophthalmology benchmark FLAIR + FIVES, PROTON improves MCM by +23.9 AUROC on covariate shift, +8.8 on semantic shift, and +8.1 on far-OOD, making it the only zero-shot method to improve all three without hierarchical prompts or labeled data. Code is available at https://github.com/GenMI-Lab/PROTON, and the project page is available at https://genmi-lab.github.io/PROTON.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 17 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

15 of 17 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Ultra-Light Test-Time Adaptation for Vision--Language Models12 Nov
  2. 2025Generalizing vision-language models to novel domains: A comprehensive survey23 Jun
  3. 2025Delving into Out-of-Distribution Detection with Medical Vision-Language Models2 Mar
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar

Other references · 11

  1. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  2. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  3. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  4. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  5. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
  6. 2025Self-Challenging Language Model Agentscited by 59
  7. 2024UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalitiescited by 46
  8. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  9. 2025OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detectioncited by 14
  10. 2025Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detectioncited by 2
  11. 2026OODBench: Out-of-Distribution Benchmark for Large Vision-Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Layer Selection in VLMs for Zero-Shot OOD Detection via Multi-Resolution Entropy Estimation
  2. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
  3. 2026TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models
  4. 2025SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Models
  5. 2026When Confidence Lacks Concepts: Interpretable OOD Detection via Representation Perturbations
  6. 2026Training-Free Out-of-Distribution Detection for Pathology Whole-Slide Images
  7. 2026Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models
  8. 2026Wasserstein-Aligned Localisation for VLM-Based Distributional OOD Detection in Medical Imaging