# Mitigating Object Hallucination via Robust Local Perception Search

arXiv: https://arxiv.org/abs/2506.06729 · HTML: https://mlsanity.com/paper/2506.06729/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 7 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing](https://mlsanity.com/paper/2404.12253/) (2024)
- [MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://mlsanity.com/paper/2401.02906/) (2024)
- [Common 7B Language Models Already Possess Strong Math Capabilities](https://mlsanity.com/paper/2403.04706/) (2024)
- [Rethinking Model Ensemble in Transfer-based Adversarial Attacks](https://mlsanity.com/paper/2303.09105/) (2023)
- [MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2411.15296/) (2024)
- [ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time](https://mlsanity.com/paper/2410.06625/) (2024)
- [Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs](https://mlsanity.com/paper/2411.09968/) (2024)
- [CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models](https://mlsanity.com/paper/2406.01920/) (2024)
- [Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models](https://mlsanity.com/paper/2405.19262/) (2024)
- [Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension](https://mlsanity.com/paper/2412.03704/) (2024)
- [LOVA3: Learning to Visual Question Answering, Asking and Assessment](https://mlsanity.com/paper/2405.14974/) (2024)
- [Inference-Time Language Model Alignment via Integrated Value Guidance](https://mlsanity.com/paper/2409.17819/) (2024)
- [Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding](https://mlsanity.com/paper/2412.06474/) (2024)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)

## Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have enabled them to effectively integrate vision and language, addressing a variety of downstream tasks. However, despite their significant success, these models still exhibit hallucination phenomena, where the outputs appear plausible but do not align with the content of the images. To mitigate this issue, we introduce Local Perception Search (LPS), a decoding method during inference that is both simple and training-free, yet effectively suppresses hallucinations. This method leverages local visual prior information as a value function to correct the decoding process. Additionally, we observe that the impact of the local visual prior on model performance is more pronounced in scenarios with high levels of image noise. Notably, LPS is a plug-and-play approach that is compatible with various models. Extensive experiments on widely used hallucination benchmarks and noisy data demonstrate that LPS significantly reduces the incidence of hallucinations compared to the baseline, showing exceptional performance, particularly in noisy settings.