mlsanityData to 14 Oct 2026

Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding

arXiv 2510.14304 ↗First posted 16 Oct 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have recently shown promising results on various multimodal tasks, even achieving human-comparable performance in certain cases. Nevertheless, LVLMs remain prone to hallucinations -- they often rely heavily on a single modality or memorize training data without properly grounding their outputs. To address this, we propose a training-free, tri-layer contrastive decoding with watermarking, which proceeds in three steps: (1) select a mature layer and an amateur layer among the decoding layers, (2) identify a pivot layer using a watermark-related question to assess whether the layer is visually well-grounded, and (3) apply tri-layer contrastive decoding to generate the final output. Experiments on public benchmarks such as POPE, MME and AMBER demonstrate that our method achieves state-of-the-art performance in reducing hallucinations in LVLMs and generates more visually grounded responses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 42 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

28 of 42 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding1 Mar
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  3. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  9. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov

Other references · 19

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
7 more
  1. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  2. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  3. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  4. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 29
  5. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  6. 2023MMVP: Motion-Matrix-based Video Predictioncited by 18
  7. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
  2. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  3. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  4. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
  5. 2026YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models
  6. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  7. 2025Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -
  8. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models