mlsanityData to 7 Oct 2026

Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models

arXiv 2406.16449 ↗First posted 24 Jun 2024cs.CVcited by 27

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

The issue of hallucinations is a prevalent concern in existing Large Vision-Language Models (LVLMs). Previous efforts have primarily focused on investigating object hallucinations, which can be easily alleviated by introducing object detectors. However, these efforts neglect hallucinations in inter-object relationships, which is essential for visual comprehension. In this work, we introduce R-Bench, a novel benchmark for evaluating Vision Relationship Hallucination. R-Bench features image-level questions that focus on the existence of relationships and instance-level questions that assess local visual comprehension. We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object. The visual instruction tuning dataset's long-tail distribution significantly impacts LVLMs' understanding of visual relationships. Furthermore, our analysis reveals that current LVLMs tend to disregard visual content and overly rely on the common sense knowledge of Large Language Models. They also struggle with reasoning about spatial relationships based on contextual information.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 56 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

48 of 56 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  3. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  4. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  5. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  6. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  8. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  9. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  10. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
24 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2016A Diagram Is Worth A Dozen Imagescited by 793
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2018nocaps: novel object captioning at scalecited by 444
  13. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  14. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  15. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  16. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  17. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  18. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  19. 2023FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsingcited by 50
  20. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  21. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23
  22. 2023Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detectioncited by 22
  23. 2023Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objectscited by 17
  24. 2023KNVQA: A Benchmark for evaluation knowledge-based VQAcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 7.

27 papers.

Sort:

By area:

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  2. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  3. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Modelscited by 30
  4. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  5. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  6. 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Frameworkcited by 17
  7. 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMscited by 14
  8. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?cited by 13
  9. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 13
  10. 2025HalLoc: Token-level Localization of Hallucinations for Vision Language Modelscited by 9
  11. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Promptscited by 5
  12. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMscited by 4
15 more
  1. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Modelscited by 2
  2. 2025Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignmentcited by 2
  3. 2026ViSA-Enhanced Aerial VLN: A Visual-Spatial Reasoning Enhanced Framework for Aerial Vision-Language Navigationcited by 2
  4. 2025What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models?cited by 1
  5. 2025Unveiling the Response of Large Vision-Language Models to Visually Absent Tokenscited by 1
  6. 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinationscited by 1
  7. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  8. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  9. 2025HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioningcited by 0
  10. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  11. 2026BABE: Biology Arena BEnchmarkcited by 0
  12. 2026Revealing Multi-View Hallucination in Large Vision-Language Modelscited by 0
  13. 2026When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noisecited by 0
  14. 2026Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactionscited by 0
  15. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding and Evaluating Hallucinations in 3D Visual Language Models
  2. 2024Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models
  3. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  4. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  5. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediations
  6. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models
  7. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
  8. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models