mlsanityData to 14 Oct 2026

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

arXiv 2606.10066 ↗First posted 8 Jun 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years, yet reported accuracy assumes these examples were absent from pretraining. We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families: image-side near-neighbour overlap against PMC-OA-beta, canonical-order exchangeability, cohort-relative Min-K%++ tail enrichment, and cross-model top-K overlap. We find measurable image-side source overlap on SLAKE-En: 19.8% of images are flagged under SigLIP-B-16 and 4.2% under SigLIP-SO400M, while out-of-domain controls produce 0/2000 flags. Manual adjudication shows same-modality, same-projection matches to different patients rather than verified pixel-level duplicates, so we interpret this as source or distributional overlap rather than confirmed per-image memorization. On the text side, Qwen2.5-VL on SLAKE-En shows a canonical-order exchangeability signal that survives ordering ablation and external non-medical baselines. On the OmniMedVQA mirror, exchangeability fires for five medical and general VLMs while BLIP-2 remains clean. In contrast, cohort-relative Min-K%++ tail enrichment and cross-model top-K overlap collapse under an external pre-domain baseline: BLIP-2 reproduces the apparent positive signals despite lacking plausible medical-VQA exposure. We conclude that these cohort-relative detectors are unreliable as standalone membership-inference signals on small medical-VLM cohorts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 24 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

23 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb

Other references · 22

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2021Membership Inference Attacks From First Principlescited by 878
  9. 2023Extracting Training Data from Diffusion Modelscited by 732
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  11. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
10 more
  1. 2025MedGemma Technical Reportcited by 436
  2. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  3. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  4. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  5. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  6. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
  7. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  8. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 81
  9. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMcited by 10
  10. 2025DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoorscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models
  2. 2026To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
  3. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  6. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
  7. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  8. 2026Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias