mlsanityData to 14 Oct 2026

ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

arXiv 2609.15635 ↗First posted 14 Sep 2026cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image. ModaLens, a paired image-swap audit, measures how report availability changes image sensitivity: MedGemma-27B on 3,199 paired MIMIC-CXR cases from 293 patients, all 14 questions per case (13 finding-specific and one composite), each image replaced by one from another study, usually of the same patient, with question and report fixed. Under an explicit answer instruction, the model's generated answer changes on 4.26 percent of trials with the report and 20.94 percent without it, a paired increase of 16.7 points (patient-clustered 95 percent CI 15.6 to 17.7), so report availability reduces image-swap sensitivity under this protocol; the original prompt with a lowercase first-token readout gives 4.70 percent against 17.07 percent, and substitutions also move continuous answer scores where the binary prediction does not change. The labels are derived from reports, which limits conclusions about visual correctness; the direction replicates in two further model lineages. Code, the exact prompts and a run record for every number are at https://github.com/criticaldata/MODALENS.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

24 of 33 references.

References · 24

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2023Steering Language Models With Activation Engineeringcited by 922
  11. 2017Selective Classification for Deep Neural Networkscited by 852
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
12 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  5. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  7. 2025MedGemma Technical Reportcited by 436
  8. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  9. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  10. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  11. 2026Vision-language models for chest radiography do not always need the imagecited by 2
  12. 2026Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AIcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unmentioned Checklist Findings Change How Reinforcement Learning Appears to Improve Chest Radiograph Report Checkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-modal linkage risk in clinical vision-language models
  2. 2026Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
  3. 2026MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
  4. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
  5. 2026PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models
  6. 2026Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
  7. 2026From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training
  8. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models