mlsanityData to 7 Oct 2026

Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding

arXiv 2605.06679 ↗First posted 22 Apr 2026cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-Language Models (VLMs) are frequently undermined by object hallucination, generating content that contradicts visual reality, due to an over-reliance on linguistic priors. We introduce Positive-and-Negative Decoding (PND), a training-free inference framework that intervenes directly in the decoding process to enforce visual fidelity. PND is motivated by our finding of an attention imbalance in VLMs, where visual features are under-weighted. Our framework introduces a dual-path contrast: a positive path that amplifies visual evidence and a negative path that constructs counterfactuals to penalize prior-dominant generation. By contrasting outputs from both paths during decoding, PND steers generation toward visually grounded results. Experiments on POPE, MME, and CHAIR demonstrate state-of-the-art performance without retraining.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 45 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

33 of 45 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models17 Mar
  2. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  3. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models30 Jun
  4. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  5. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  6. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  10. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  11. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 22

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
10 more
  1. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  7. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  8. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  9. 2024Streamlining Prediction in Bayesian Deep Learningcited by 5
  10. 2022Recent Advances in Algebraic Geometry and Bayesian Statisticscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
  2. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models
  3. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
  4. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
  5. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  6. 2026See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
  7. 2025CoFi-Dec: Hallucination-Resistant Decoding via Coarse-to-Fine Generative Feedback in Large Vision-Language Models
  8. 2026CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs