mlsanityData to 14 Oct 2026

Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models

arXiv 2402.15721 ↗First posted 24 Feb 2024cs.AI cs.CLcited by 27

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision Language Models exhibit remarkable capabilities but struggle with hallucinations inconsistencies between images and their descriptions. Previous hallucination evaluation studies on LVLMs have identified hallucinations in terms of objects, attributes, and relations but overlooked complex hallucinations that create an entire narrative around a fictional entity. In this paper, we introduce a refined taxonomy of hallucinations, featuring a new category: Event Hallucination. We then utilize advanced LLMs to generate and filter fine grained hallucinatory data consisting of various types of hallucinations, with a particular focus on event hallucinations, laying the groundwork for integrating discriminative and generative evaluation methods within our universal evaluation framework. The proposed benchmark distinctively assesses LVLMs ability to tackle a broad spectrum of hallucinations, making it a reliable and comprehensive tool for gauging LVLMs efficacy in handling hallucinations. We will release our code and data.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 50 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

35 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  2. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  3. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  4. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 31

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
19 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2018Object Hallucination in Image Captioningcited by 669
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  16. 2023TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selectioncited by 18
  17. 2023BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarizationcited by 5
  18. 2023COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignmentcited by 5
  19. 2023TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-trainingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 11; 2026: 5.

27 papers.

Sort:

By area:

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  2. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 60
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  4. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 32
  5. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 32
  6. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  7. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 30
  8. 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 30
  9. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  10. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  11. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  12. 2024Interpreting and Mitigating Hallucination in MLLMs through Multi-agent Debatecited by 16
15 more
  1. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 15
  2. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  3. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 12
  4. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  5. 2024NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queriescited by 10
  6. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
  7. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  8. 2024FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphscited by 3
  9. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  10. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  11. 2026Said Aloud, Read Different: Cross-Modal Instability in Multimodal Modelscited by 2
  12. 2025HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restorationcited by 1
  13. 2025Value-Guided Iterative Refinement and the DIQ-H Benchmark for Evaluating VLM Robustnesscited by 1
  14. 2026Attention at Rest Stays at Rest: Breaking Visual Inertia for Cognitive Hallucination Mitigationcited by 0
  15. 2026Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Hallucination in Large Vision-Language Models
  2. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
  3. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  4. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models
  6. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models
  7. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models
  8. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities