Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Reliable deployment of Vision-Language Models (VLMs) in radiology requires validation metrics that go beyond surface-level text similarity to ensure clinical fidelity and demographic fairness. This paper investigates a critical blind spot in current model evaluation: the use of decoding strategies that lead to high aggregate token-overlap scores despite succumbing to template collapse, in which models generate only repetitive, safe generic text and omit clinical terminology. Unaddressed, this blind spot can lead to metric gaming, where models that perform well on benchmarks prove clinically uninformative. Instead, we advocate for lexical diversity measures to check model generations for clinical specificity. We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports. Weighted Association Erasure (WAE) aggregates these shifts to measure the clinical signal loss across demographic groups. We show that deterministic decoding produces high levels of semantic erasure, while stochastic sampling generates diverse outputs but risks introducing new bias, motivating a fundamental rethink of how "optimal" reporting is defined.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 30 references that are in the map. First posted on arXiv 2 Mar 2026.
Builds on
14 of 30 references.
Closest ancestors · 1
Same direction, newest first
Other references · 13
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
- 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
- 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
- 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
- 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
- 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
- 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
- 2025The Price of Format: Diversity Collapse in LLMscited by 43
- 2025ReXGradient-160K: A Large-Scale Publicly Available Dataset of Chest Radiographs with Free-text Reportscited by 24
- 2025Investigating Label Bias and Representational Sources of Age-Related Disparities in Medical Segmentationcited by 5
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
- 2026Improving Health Literacy through Lay Summarization of Radiological Reports: An Evaluation of BioNER and Retrieval-Augmented Generation
- 2026Medical Context Distorts Decisions in Clinical Vision Language Models
- 2026CCS: Clinical Consensus Selection for Radiology Report Generation
- 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
- 2025ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment
- 2026Cross-modal linkage risk in clinical vision-language models
- 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation