mlsanityData to 7 Oct 2026

IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding

arXiv 2402.18476 ↗First posted 28 Feb 2024cs.CVcited by 118

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite achieving rapid developments and with widespread applications, Large Vision-Language Models (LVLMs) confront a serious challenge of being prone to generating hallucinations. An over-reliance on linguistic priors has been identified as a key factor leading to these hallucinations. In this paper, we propose to alleviate this problem by introducing a novel image-biased decoding (IBD) technique. Our method derives the next-token probability distribution by contrasting predictions from a conventional LVLM with those of an image-biased LVLM, thereby amplifying the correct information highly correlated with image content while mitigating the hallucinatory errors caused by excessive dependence on text. We further conduct a comprehensive statistical analysis to validate the reliability of our method, and design an adaptive adjustment strategy to achieve robust and flexible handling under varying conditions. Experimental results across multiple evaluation metrics verify that our method, despite not requiring additional training data and only with a minimal increase in model parameters, can significantly reduce hallucinations in LVLMs and enhance the truthfulness of the generated response.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 48 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

41 of 48 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  2. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  6. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  7. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  8. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  9. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  10. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  11. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2018Hierarchical Neural Story Generationcited by 1,182
18 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  2. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  10. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  11. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  12. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  13. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  14. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  15. 2023LLaFS: When Large Language Models Meet Few-Shot Segmentationcited by 48
  16. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  17. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  18. 2023Pixel Aligned Language Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 45; 2026: 42.

118 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  2. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  3. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  4. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  5. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  6. 2024SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and Morecited by 58
  7. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  8. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  9. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  10. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  11. 2024Probing Visual Language Priors in VLMscited by 44
  12. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
38 more
  1. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  2. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  3. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  4. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  5. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  6. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  7. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  8. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27
  9. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  10. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 21
  11. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  12. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understandingcited by 20
  13. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  14. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignmentcited by 19
  15. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  16. 2024Discrete Latent Perspective Learning for Segmentation and Detectioncited by 16
  17. 2024Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Modelscited by 14
  18. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
  19. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Modelscited by 12
  20. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decodingcited by 12
  21. 2024Unveiling and Mitigating Bias in Audio Visual Segmentationcited by 11
  22. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 11
  23. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
  24. 2024PPTFormer: Pseudo Multi-Perspective Transformer for UAV Segmentationcited by 10
  25. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  26. 2025Structural and Statistical Texture Knowledge Distillation and Learning for Segmentationcited by 9
  27. 2024Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decodingcited by 8
  28. 2024TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddingscited by 8
  29. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
  30. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  31. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  32. 2026VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneckcited by 8
  33. 2024VACoDe: Visual Augmented Contrastive Decodingcited by 7
  34. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  35. 2025Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbationscited by 7
  36. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
  37. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  38. 2025SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentationcited by 7

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  3. 2025Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -
  4. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  5. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
  6. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  7. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)
  8. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance