# Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2408.09429 · HTML: https://mlsanity.com/paper/2408.09429/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web](https://mlsanity.com/paper/2310.18340/) (2023)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Word Embeddings Are Steers for Language Models](https://mlsanity.com/paper/2305.12798/) (2023)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2406.16449/) (2024)

## Built on by (most cited first, 30 of 30)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2410.03577/) (2024)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention](https://mlsanity.com/paper/2510.02912/) (2025)
- [Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality](https://mlsanity.com/paper/2410.04780/) (2024)
- [ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection](https://mlsanity.com/paper/2410.04509/) (2024)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection](https://mlsanity.com/paper/2505.13312/) (2025)
- [GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning](https://mlsanity.com/paper/2508.04088/) (2025)
- [CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2510.12184/) (2025)
- [ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models](https://mlsanity.com/paper/2409.09318/) (2024)
- [Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models](https://mlsanity.com/paper/2410.23114/) (2024)
- [Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation](https://mlsanity.com/paper/2602.24041/) (2026)
- [Measuring Epistemic Humility in Multimodal Large Language Models](https://mlsanity.com/paper/2509.09658/) (2025)
- [CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models](https://mlsanity.com/paper/2601.00659/) (2026)
- [A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models](https://mlsanity.com/paper/2603.29676/) (2026)
- [Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning](https://mlsanity.com/paper/2606.00105/) (2026)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue](https://mlsanity.com/paper/2606.00622/) (2026)
- [ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations](https://mlsanity.com/paper/2510.06292/) (2025)
- [Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time](https://mlsanity.com/paper/2605.01766/) (2026)
- [When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs](https://mlsanity.com/paper/2605.11559/) (2026)
- [Spatial Knowledge Graph-Guided Multimodal Synthesis](https://mlsanity.com/paper/2505.22633/) (2025)
- [Attention at Rest Stays at Rest: Breaking Visual Inertia for Cognitive Hallucination Mitigation](https://mlsanity.com/paper/2604.01989/) (2026)
- [When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise](https://mlsanity.com/paper/2605.05045/) (2026)
- [Consistency as Inductive Bias: Learning Cross-View Invariance for Robust Multimodal Reasoning](https://mlsanity.com/paper/2606.29812/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads](https://mlsanity.com/paper/2609.09206/) (2026)

## Abstract (arXiv)

Hallucination issues continue to affect multimodal large language models (MLLMs), with existing research mainly addressing object-level or attribute-level hallucinations, neglecting the more complex relation hallucinations that require advanced reasoning. Current benchmarks for relation hallucinations lack detailed evaluation and effective mitigation, and their datasets often suffer from biases due to systematic annotation processes. To address these challenges, we introduce Reefknot, a comprehensive benchmark targeting relation hallucinations, comprising over 20,000 real-world samples. We provide a systematic definition of relation hallucinations, integrating perceptive and cognitive perspectives, and construct a relation-based corpus using the Visual Genome scene graph dataset. Our comparative evaluation reveals significant limitations in current MLLMs' ability to handle relation hallucinations. Additionally, we propose a novel confidence-based mitigation strategy, which reduces the hallucination rate by an average of 9.75% across three datasets, including Reefknot. Our work offers valuable insights for achieving trustworthy multimodal intelligence.