mlsanityData to 14 Oct 2026

Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMs

arXiv 2512.17189 ↗First posted 19 Dec 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Medical Vision-Language Models (MedVLMs) show immense promise in clinical applicability. However, their reliability is hindered by hallucinations, where models often fail to derive answers from visual evidence, instead relying on learned textual priors. Existing mitigation strategies for MedVLMs have distinct limitations: training-based methods rely on costly expert annotations, limiting scalability, while training-free interventions like contrastive decoding, though data-efficient, apply a global, untargeted correction whose effects in complex real-world clinical settings can be unreliable. To address these challenges, we introduce Anatomical Region-Guided Contrastive Decoding (ARCD), a plug-and-play strategy that mitigates hallucinations by providing targeted, region-specific guidance. Our module leverages an anatomical mask to direct a three-tiered contrastive decoding process. By dynamically re-weighting at the token, attention, and logits levels, it verifiably steers the model's focus onto specified regions, reinforcing anatomical understanding and suppressing factually incorrect outputs. Extensive experiments across diverse datasets, including chest X-ray, CT, brain MRI, and ocular ultrasound, demonstrate our method's effectiveness in improving regional understanding, reducing hallucinations, and enhancing overall diagnostic accuracy.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 33 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

24 of 33 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding10 Jun
  2. 2025Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training17 Apr
  3. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  4. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
  5. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov

Other references · 17

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  6. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  7. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  8. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  9. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  10. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  11. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  12. 2024STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answeringcited by 21
5 more
  1. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  2. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  3. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationalecited by 14
  4. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbookscited by 14
  5. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Studycited by 1
  2. 2026Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
  2. 2025MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
  3. 2026VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs
  4. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  5. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  6. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context
  7. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  8. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance