mlsanityData to 7 Oct 2026

PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset

arXiv 2403.11116 ↗First posted 17 Mar 2024cs.CV cs.AIcited by 44

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) hallucinate, resulting in an emerging topic of visual hallucination evaluation (VHE). This paper contributes a ChatGPT-Prompted visual hallucination evaluation Dataset (PhD) for objective VHE at a large scale. The essence of VHE is to ask an MLLM questions about specific images to assess its susceptibility to hallucination. Depending on what to ask (objects, attributes, sentiment, etc.) and how the questions are asked, we structure PhD along two dimensions, i.e. task and mode. Five visual recognition tasks, ranging from low-level (object / attribute recognition) to middle-level (sentiment / position recognition and counting), are considered. Besides a normal visual QA mode, which we term PhD-base, PhD also asks questions with specious context (PhD-sec) or with incorrect context ({PhD-icc), or with AI-generated counter common sense images (PhD-ccs). We construct PhD by a ChatGPT-assisted semi-automated pipeline, encompassing four pivotal modules: task-specific hallucinatory item (hitem) selection, hitem-embedded question generation, specious / incorrect context generation, and counter-common-sense (CCS) image generation. With over 14k daily images, 750 CCS images and 102k VQA triplets in total, PhD reveals considerable variability in MLLMs' performance across various modes and tasks, offering valuable insights into the nature of hallucination. As such, PhD stands as a potent tool not only for VHE but may also play a significant role in the refinement of MLLMs.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 43 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

29 of 43 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  3. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges6 Nov
  4. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  6. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  7. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  8. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  9. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  10. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
7 more
  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  3. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  4. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  5. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  6. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
  7. 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sensecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 11; 2026: 23.

44 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  2. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  4. 2025Vision Language Models are Biasedcited by 63
  5. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  6. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  7. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  8. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  9. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  10. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  11. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8
  12. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
32 more
  1. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  2. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
  3. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  4. 2026V-FAT: Benchmarking Visual Fidelity Against Text-biascited by 6
  5. 2024Harnessing GPT-4V(ision) for Insurance: A Preliminary Explorationcited by 5
  6. 2025Empowering Small VLMs to Think with Dynamic Memorization and Explorationcited by 5
  7. 2025TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detectioncited by 4
  8. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  9. 2025Two-Stage Regularization-Based Structured Pruning for LLMscited by 3
  10. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidencecited by 3
  11. 2026Once Correct, Still Wrong: Counterfactual Hallucination in Multilingual Vision-Language Modelscited by 2
  12. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  13. 2025C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluationcited by 1
  14. 2025SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMscited by 1
  15. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  16. 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoningcited by 1
  17. 2026GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinationscited by 1
  18. 2026EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectorscited by 1
  19. 2026Faithful Mobile GUI Agents with Guided Advantage Estimatorcited by 1
  20. 2026ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluationcited by 1
  21. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1
  22. 2026KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluationcited by 1
  23. 2026Unified Hallucination Fuzzing for Multimodal Large Language Modelscited by 1
  24. 2024Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructionscited by 0
  25. 2026TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuningcited by 0
  26. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  27. 2026Prior Bias in Vision Language Models on UML Diagram Interpretationcited by 0
  28. 2026Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilitiescited by 0
  29. 2026Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentationcited by 0
  30. 2026UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Spacecited by 0
  31. 2026SABRE: Scalable and Automated Benchmarking of VLMs under Stresscited by 0
  32. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediations
  2. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
  3. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context
  4. 2024Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models
  5. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  6. 2024FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs
  7. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
  8. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback