mlsanityData to 7 Oct 2026

Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation

arXiv 2605.27993 ↗First posted 27 May 2026cs.CLcited by 0

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Object hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume hallucinations stem from visual neglect, steering models to enhance visual reliance. In contrast, our systematic interventions on multiple MLLMs show that pushing toward more visual reliance may exacerbate hallucinations on some models, while less may mitigate hallucinations. This result suggests that attributing hallucinations solely to visual insufficiency is underdetermined. We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context. For this, we propose a training-free framework, Context-Preference Activation Steering (CAS). It extracts two semantically distinct Context Preference Vectors (CPVs) via two small sets of designed conflict samples and applies them via single-pass signed residual injection at mid-early MLP layers during inference to control information reliance. Experiments show that CAS substantially mitigates object hallucinations without increasing decoding latency and preserves native text-generation quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 26 of its 32 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

26 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models9 Apr

Other references · 25

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  11. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
13 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  3. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  4. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  5. 2025ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLMcited by 36
  6. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  7. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  8. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 9
  9. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  10. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  11. 2025Steering LVLMs via Sparse Autoencoder for Hallucination Mitigationcited by 6
  12. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 3
  13. 2025V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Interventioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
  2. 2025Mitigating Object Hallucinations via Sentence-Level Early Intervention
  3. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  4. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
  5. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
  6. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization
  7. 2025Stop learning it all to mitigate visual hallucination, Focus on the hallucination target
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens