mlsanityData to 14 Oct 2026

Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models

arXiv 2609.32185 ↗First posted 26 Sep 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Pathology vision-language models (VLMs) are conventionally evaluated by accuracy, but accuracy alone does not measure evidence use: it may conflate dataset contamination, prior knowledge, and image evidence. In a motivating study of lymph-node metastasis prediction, we found that most public pathology VLMs showed minimal differences when changing from feeding the models with whole-slide images, an annotated lesion, or no image at all. To better understand the specific features leveraged by these models, this paper presents two contributions aimed at disentangling these factors. First, we present CleanSlide, a TCGA-based VQA benchmark designed to eliminate image- and question-side contamination. It contains 149K audited multiple-choice questions over 9,985 slides, with patient- and tissue-source-disjoint splits. Every question is audited for option shortcuts, stem leakage, cross-split duplication, and blind solvability. Second, we propose Pair-DPO, a preference loss over counterfactual slide pairs from the same question and source. By controlling for shared confounding factors, Pair-DPO cancels out the question-attributable signal and leaves image evidence as the source of preference. Specifically, each pair consists of two real slides with opposite, verified findings, introducing neither editing artifacts nor unverified labels for diffuse or graded features such as invasion, necrosis, and tumor grade. Experiments show that our method gains 15.29% from image evidence on the CleanSlide, compared with 2.81% for the best published model. On the external CPTAC and BCNB cohorts, our method achieves accuracies of 57.6% and 59.0%, outperforming all other evaluated models by 9.7% and 3.4%, respectively. We will release the benchmark and code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 61 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

31 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks14 Jul
  2. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning4 Aug
  3. 2025A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model23 Jul
  4. 2025Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner16 May
  5. 2025PathVLM-R1: A Reinforcement Learning-Driven Reasoning Model for Pathology Visual-Language Tasks12 Apr
  6. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning26 Feb
  7. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology16 Dec
  8. 2024WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image3 Dec
  9. 2024SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding15 Oct

Other references · 22

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  10. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  11. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  12. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
10 more
  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  2. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  3. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  4. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  5. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  6. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 85
  7. 2025PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathologycited by 39
  8. 2025CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logiccited by 29
  9. 2025WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image Analysiscited by 7
  10. 2026Enhancing Pathological VLMs with Cross-scale Reasoningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Democratising Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modelling
  2. 2026A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks
  3. 2026Do Pathology Vision-Language Models Truly See Pathology?
  4. 2026Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
  5. 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
  6. 2026PathGLS: Evaluating Pathology Vision-Language Models without Ground Truth through Multi-Dimensional Consistency
  7. 2026PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology
  8. 2026EviPathBench: Benchmarking Evidence Acquisition and Reasoning in Vision-Language Models for Whole-Slide Pathology