mlsanityData to 7 Oct 2026

PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models

arXiv 2602.21428 ↗First posted 24 Feb 2026cs.CV cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical Vision Language Models (VLMs) can change their answers when clinicians rephrase the same question, a failure mode that threatens deployment safety. We introduce PSF-Med, a benchmark of 26,850 chest X-ray questions paired with 92,856 meaning-preserving paraphrases across MIMIC-CXR, PadChest, and VinDr-CXR, spanning clinical populations in the US, Spain, and Vietnam. Every paraphrase is validated by an LLM judge using a bidirectional clinical entailment rubric, with 91.6% cross-family agreement. Across nine VLMs, including general-purpose models, we find flip rates from 3% to 37%. However, low flip rate does not imply visual grounding: text-only baselines show that some models stay consistent even when the image is removed, suggesting they rely on language priors. To study mechanisms in one model, we apply GemmaScope 2 Sparse Autoencoders (SAEs) to MedGemma 4B and analyze FlipBank, a curated set of 158 flip cases. We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts. In causal patching, removing this feature's contribution recovers 45% of the yes-minus-no logit margin on average and fully reverses 15% of flips. Acting on this finding, we show that clamping the identified feature at inference reduces flip rates by 31% relative with only a 1.3 percentage-point accuracy cost, while also decreasing text-prior reliance. These results suggest that flip rate alone is not enough; robustness evaluations should test both paraphrase stability and image reliance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 38 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

27 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2019Publicly Available Clinical BERT Embeddingscited by 995
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  9. 2019Attention is not Explanationcited by 916
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2019Attention is not not Explanationcited by 629
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
14 more
  1. 2021Causal Abstractions of Neural Networkscited by 401
  2. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  3. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  4. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  5. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  6. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  7. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  8. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  9. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  10. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  11. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  12. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  13. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  14. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Modelscited by 2
  2. 2026Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMscited by 1
  3. 2026Lightweight Stylistic Consistency Profiling: Robust Detection of LLM-Generated Textual Content for Multimedia Moderationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
  2. 2026Medical Context Distorts Decisions in Clinical Vision Language Models
  3. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
  4. 2026Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models
  5. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  6. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
  7. 2026To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
  8. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering