mlsanityData to 14 Oct 2026

ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models

arXiv 2503.13107 ↗First posted 17 Mar 2025cs.CV cs.AIcited by 40

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Contrastive decoding strategies are widely used to mitigate object hallucinations in multimodal large language models (MLLMs). By reducing over-reliance on language priors, these strategies ensure that generated content remains closely grounded in visual inputs, producing contextually accurate outputs. Since contrastive decoding requires no additional training or external tools, it offers both computational efficiency and versatility, making it highly attractive. However, these methods present two main limitations: (1) bluntly suppressing language priors can compromise coherence and accuracy of generated content, and (2) processing contrastive inputs adds computational load, significantly slowing inference speed. To address these challenges, we propose Visual Amplification Fusion (VAF), a plug-and-play technique that enhances attention to visual signals within the model's middle layers, where modality fusion predominantly occurs. This approach enables more effective capture of visual features, reducing the model's bias toward language modality. Experimental results demonstrate that VAF significantly reduces hallucinations across various MLLMs without affecting inference speed, while maintaining coherence and accuracy in generated outputs.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 54 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

45 of 54 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  3. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  9. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
24 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  10. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  11. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  12. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  13. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  14. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  15. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  16. 2023Embodied Task Planning with Large Language Modelscited by 84
  17. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  18. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  19. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 33
  20. 2023Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPTcited by 28
  21. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  22. 2023LLM-Based Human-Robot Collaboration Framework for Manipulation Taskscited by 17
  23. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16
  24. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 30.

40 papers.

Sort:

By area:

  1. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  2. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  3. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  4. 2026Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examinationcited by 13
  5. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 12
  6. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  7. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  8. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 5
  9. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Modelscited by 4
  10. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 4
  11. 2025SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketchescited by 3
  12. 2025V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Interventioncited by 3
28 more
  1. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 3
  2. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  3. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Modelscited by 2
  4. 2025GeoPE:A Unified Geometric Positional Embedding for Structured Tensorscited by 2
  5. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  6. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  7. 2026EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Modelscited by 2
  8. 2026HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Modelscited by 2
  9. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  10. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  11. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancementcited by 1
  12. 2026Breaking the Illusion: When Positive Meets Negative in Multimodal Decodingcited by 1
  13. 2026Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approachcited by 1
  14. 2026VERHallu: Evaluating and Mitigating Event Relation Hallucination in Video Large Language Modelscited by 0
  15. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMscited by 0
  16. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
  17. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  18. 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchorscited by 0
  19. 2026Attention at Rest Stays at Rest: Breaking Visual Inertia for Cognitive Hallucination Mitigationcited by 0
  20. 2026See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignmentcited by 0
  21. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  22. 2026Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agentcited by 0
  23. 2026Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosiscited by 0
  24. 2026Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignmentcited by 0
  25. 2026Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Modelscited by 0
  26. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0
  27. 2026DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editingcited by 0
  28. 2026Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgerycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  2. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
  3. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
  4. 2026Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decoding
  5. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  6. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  7. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  8. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance