mlsanityData to 7 Oct 2026

FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs

arXiv 2409.13612 ↗First posted 20 Sep 2024cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

The rapid development of Large Vision-Language Models (LVLMs) often comes with widespread hallucination issues, making cost-effective and comprehensive assessments increasingly vital. Current approaches mainly rely on costly annotations and are not comprehensive -- in terms of evaluating all aspects such as relations, attributes, and dependencies between aspects. Therefore, we introduce the FIHA (autonomous Fine-graIned Hallucination evAluation evaluation in LVLMs), which could access hallucination LVLMs in the LLM-free and annotation-free way and model the dependency between different types of hallucinations. FIHA can generate Q&A pairs on any image dataset at minimal cost, enabling hallucination assessment from both image and caption. Based on this approach, we introduce a benchmark called FIHA-v1, which consists of diverse questions on various images from MSCOCO and Foggy. Furthermore, we use the Davidson Scene Graph (DSG) to organize the structure among Q&A pairs, in which we can increase the reliability of the evaluation. We evaluate representative models using FIHA-v1, highlighting their limitations and challenges. We released our code and data.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

38 of 47 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models16 Jun
  2. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  3. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models24 Feb
  4. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  5. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  6. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  7. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges6 Nov
  8. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  9. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  10. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  11. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
13 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  7. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  8. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  9. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  10. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  11. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  12. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  13. 2024Fine-grained and Explainable Factuality Evaluation for Multimodal Summarizationcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Trust but Verify: Programmatic VLM Evaluation in the Wildcited by 5
  2. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  3. 2025GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integrationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  2. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
  3. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models
  4. 2025SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs
  5. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context
  6. 2024A Survey on Hallucination in Large Vision-Language Models
  7. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  8. 2024Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models