mlsanityData to 14 Oct 2026

Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding

arXiv 2409.06485 ↗First posted 10 Sep 2024cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Although Visual-Language Models (VLMs) have shown impressive capabilities in tasks like visual question answering and image captioning, they still struggle with hallucinations. Analysis of attention distribution in these models shows that VLMs tend to processing textual tokens rather than visual tokens. This imbalance of attention distribution causes VLMs to favor textual knowledge in the case of multimodal knowledge conflicts, resulting in differences from the image information. In this paper, we propose Re-Balancing Contrastive Decoding (RBD) method, which employs textual and visual branches to recalibrate attention distribution in VLMs. Specifically, the textual branch injects image noise to stimulate the model's dependency on text, thereby reducing textual bias. Concurrently, the visual branch focuses on the selection of significant tokens, refining the attention mechanism to highlight the primary subject. This dual-branch strategy enables the RBD method to diminish textual bias while enhancing visual information. Experimental results demonstrate that our method, RBD, outperforms the existing methods by the CHAIR and POPE metrics, mitigate hallucinations without reducing the model's general capabilities.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 42 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

37 of 42 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  6. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  7. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  8. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
17 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  2. 2022Token Merging: Your ViT But Fastercited by 1,021
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  5. 2024Yi: Open Foundation Models by 01.AIcited by 719
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  10. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  12. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  13. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  14. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 79
  15. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  16. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  17. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  2. 2025MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Modelscited by 10
  3. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  4. 2025Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuningcited by 3
  5. 2025LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehensioncited by 1
  6. 2025Enhancing Medical Large Vision-Language Models via Alignment Distillationcited by 1
  7. 2026Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoningcited by 1
  8. 2026ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
  2. 2025ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
  3. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  4. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  5. 2026Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation
  6. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
  7. 2026Revealing Multi-View Hallucination in Large Vision-Language Models
  8. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding