mlsanityData to 7 Oct 2026

AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation

arXiv 2311.07397 ↗First posted 13 Nov 2023cs.CL cs.CVcited by 293

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite making significant progress in multi-modal tasks, current Multi-modal Large Language Models (MLLMs) encounter the significant challenge of hallucinations, which may lead to harmful consequences. Therefore, evaluating MLLMs' hallucinations is becoming increasingly important in model improvement and practical application deployment. Previous works are limited in high evaluation costs (e.g., relying on humans or advanced LLMs) and insufficient evaluation dimensions (e.g., types of tasks and hallucinations). In this paper, we propose an LLM-free multi-dimensional benchmark AMBER, which can be used to evaluate both generative task and discriminative task including existence, attribute and relation hallucination. Based on AMBER, we design a low-cost and efficient evaluation pipeline. Additionally, we conduct a comprehensive evaluation and detailed analysis of mainstream MLLMs including GPT-4V(ision), and also give guideline suggestions for mitigating hallucinations. The data and code of AMBER are available at https://github.com/junyangwang0410/AMBER.

Key facts. Cited by 293 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 22 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

11 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges6 Nov

Other references · 10

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  10. 2023An Early Evaluation of GPT-4V(ision)cited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 76; 2025: 101; 2026: 114.

293 papers.

Sort:

By area:

  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  3. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  4. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  5. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  6. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  7. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  8. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  9. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  10. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  11. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  12. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
38 more
  1. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  2. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  3. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  4. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  5. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  6. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  7. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  8. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  9. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  10. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  11. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  12. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  13. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  14. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  15. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  16. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  17. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 42
  18. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  19. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 37
  20. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 37
  21. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  22. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  23. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Modelscited by 31
  24. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  25. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  26. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Modelscited by 30
  27. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  28. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  29. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  30. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27
  31. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 27
  32. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  33. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  34. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  35. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  36. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  37. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  38. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20

50 most cited of 293 All 293 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  2. 2024Lynx: An Open Source Hallucination Evaluation Model
  3. 2025MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations
  4. 2024THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models
  5. 2025Theoretical Foundations and Mitigation of Hallucination in Large Language Models
  6. 2025OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models
  7. 2024Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
  8. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models