mlsanityData to 14 Oct 2026

Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models

arXiv 2609.17269 ↗First posted 15 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal large language models generate natural-language responses from visual inputs, yet may mention objects absent from an image. In medication assistance, accessible perception, and environmental decision-making, such hallucinations can create real-world safety risks. We propose Semantic-Spatial Agreement Verification (SSAV), a training-free method for verifying object claims. A visually grounded claim should remain stable across semantically equivalent queries and repeatedly localize to the same image region. SSAV aggregates multiple prompts to estimate semantic support and reduce sensitivity to query wording. Query-Induced Regional Verification (QIRV) combines cross-query region persistence, spatial overlap, and relative candidate dominance to identify isolated high responses and dispersed localizations. A geometric mean fuses semantic and spatial evidence, lowering the verification score when either branch lacks support. Experiments on three base models and multiple evaluation protocols show that SSAV effectively mitigates object hallucination. On LLaVA-1.5-7B, accuracy averaged across COCO, A-OKVQA, and GQA improves by 1.81 and 3.17 percentage points under POPE Popular and Adversarial, respectively, while CHAIRs decreases from 49.40% to 32.80%. These results show that cross-query semantic stability and regional consistency provide interpretable external visual evidence for object claims.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 59 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

54 of 59 references.

Closest ancestors · 31

Same direction, newest first

  1. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models28 Apr
  2. 2026Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression11 Mar
  3. 2026Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models12 Feb
  4. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothing4 Feb
  5. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models10 Oct
  6. 2025Energy-Guided Decoding for Object Hallucination Mitigation10 Jul
  7. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling7 Jul
  8. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models17 Mar
  9. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention13 Mar
  10. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding13 Mar
  11. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation11 Mar
  12. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models10 Feb
19 more
  1. 2025VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification11 Jan
  2. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigation3 Jan
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  4. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
  5. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  6. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
  7. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning30 Aug
  8. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models25 Aug
  9. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  10. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  11. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  12. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  13. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  14. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  15. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  16. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  17. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  18. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  19. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2021Grounded Language-Image Pre-trainingcited by 1,156
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
11 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  2. 2018Object Hallucination in Image Captioningcited by 669
  3. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  4. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  6. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  7. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  8. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  9. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  10. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  11. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction
  2. 2026From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models
  3. 2026R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs
  4. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  5. 2026Geo3R: Mitigating Spatial Reasoning Hallucination in Multimodal Large Language Models
  6. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  7. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models
  8. 2026HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models