mlsanityData to 13 Oct 2026

From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks

arXiv 2605.24771 ↗First posted 23 May 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Classical noisy-label theory predicts that downstream performance under weak supervision is bounded above by the labeler's accuracy, implying a sharp crossover: once a gold-trained classifier matches the labeler, weak labels stop helping and start hurting. The prediction is theoretical; what is missing is a benchmark calibration that turns it into an instance-level statement for modern foundation-model labelers. We provide such a calibration for BiomedCLIP-generated weak labels on three medical-imaging benchmarks (PCAM, ISIC, NIH-CXR) and six downstream architectures spanning an 11x parameter range. The crossover predicted by theory appears at ng~100 on PCAM, 20-50 on ISIC, and 250-500 on NIH-CXR; weak labels above the crossover degrade AUC by up to -0.10. The location is architecture-invariant for four of five pretrained architectures, and a within-family DenseNet sweep (2.5x parameters, identical pretraining) supports the view that the labeler, not the student, is the dominant constraint. The calibration in turn produces a decision rule operable from 10-20 gold labels: compare gold-only AUC to VLM accuracy on the user's gold set. A structured-vs-random noise sign flip on NIH-CXR shows that the rate-only formulation of the bound is incomplete and identifies a concrete refinement (label-space projection) that future benchmarks can be designed to test.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 9 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

7 of 9 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VLM-CPL: Consensus Pseudo Labels from Vision-Language Models for Annotation-Free Pathological Image Classification23 Mar

Other references · 6

  1. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,082
  2. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,306
  3. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  4. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 581
  5. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 514
  6. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
  2. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
  3. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models
  4. 2026Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations
  5. 2026CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis
  6. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
  7. 2026Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learning
  8. 2025Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients