RECAP: Relation Evidence Calibration for Detecting Spatial Relation Hallucinations in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs
Abstract (arXiv)
Vision-language models can answer spatial relation questions confidently even when the image supports an incompatible relation. We formulate relation-grounded selective prediction: accept or reject an already-produced yes/no answer by auditing its visual support, rather than treating uncertainty as evidence. RECAP, our relation-evidence calibration framework, compares image-conditioned likelihoods for a claim, its semantic contradictions, and optional one-sided supports, then converts these witnesses into an answer-conditioned rejection risk. A calibration-only gate preserves confidence as a veto when confidence is demonstrably informative and otherwise deploys relation evidence alone. Across 20 group/image-disjoint splits, RECAP lowers H-FPR@80 over confidence by between 2.0 and 17.9 points on VSR and raises Acc@80 by 3.0, 8.6, and 12.6 points on What'sUp for Qwen3-VL-8B, InternVL3.5-8B, and LLaVA-1.5-7B. It outperforms matched VCD-style visual contrast on all four primary metrics in all six settings. Full-pool VSR fallback, target-ranked GSR-Bench transfer, equal-budget supervised controls, and two additional checkpoints show a consistent operating principle: structured counterevidence complements certainty when confidence is misaligned, while the gate retains confidence when it is already useful.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 41 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
27 of 41 references.
Closest ancestors · 9
Same direction, newest first
- 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token5 Mar
- 2025Taming Object Hallucinations with Verified Atomic Confidence Estimation12 Nov
- 2025Mitigating Hallucinations in Multimodal Spatial Relations through Constraint-Aware Prompting12 Feb
- 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation18 Nov
- 2024Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification2 Jul
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
- 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
- 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
Other references · 18
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2015VQA: Visual Question Answeringcited by 2,506
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2018Object Hallucination in Image Captioningcited by 669
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
- 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
- 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
6 more
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
- 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
- 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
- 2019SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognitioncited by 55
- 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
- 2026Predicting When to Trust Vision-Language Models for Spatial Reasoningcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise
- 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations
- 2026Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models
- 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models
- 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
- 2024MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Models
- 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models
- 2026CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generation