mlsanityData to 14 Oct 2026

Mitigating Action-Relation Hallucinations in LVLMs via Relation-aware Visual Enhancement

arXiv 2605.11808 ↗First posted 12 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved remarkable performance on diverse vision-language tasks. However, LVLMs still suffer from hallucinations, generating text that contradicts the visual input. Existing research has primarily focused on mitigating object hallucinations, but often overlooks more complex relation hallucinations, particularly action relations involving interactions between objects. In this study, we empirically observe that the primary cause of action-relation hallucinations in LVLMs is the insufficient attention allocated to visual information. Thus, we propose a framework to locate action-relevant image regions and enhance the LVLM's attention to those regions. Specifically, we define the Action-Relation Sensitivity (ARS) score to identify attention heads that are most sensitive to action-relation changes, thereby localizing action-relevant image regions that contain key visual cues. Then, we propose the Relation-aware Visual Enhancement (RVE) method to enhance the LVLM's attention to these action-relevant image regions. Extensive experiments demonstrate that, compared to existing baselines, our method achieves superior performance in mitigating action-relation hallucinations with negligible additional inference cost. Furthermore, it effectively generalizes to spatial-relation hallucinations and object hallucinations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 23 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

19 of 23 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence18 Dec
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  3. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  11. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov

Other references · 8

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  8. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  2. 2026Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
  3. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
  4. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
  5. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
  6. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  7. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  8. 2025Mitigating Hallucinations in Multimodal Spatial Relations through Constraint-Aware Prompting