mlsanityData to 7 Oct 2026

HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models

arXiv 2310.14566 ↗First posted 23 Oct 2023cs.CV cs.CLcited by 572

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

We introduce HallusionBench, a comprehensive benchmark designed for the evaluation of image-context reasoning. This benchmark presents significant challenges to advanced large visual-language models (LVLMs), such as GPT-4V(Vision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, by emphasizing nuanced understanding and interpretation of visual data. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. We introduce a novel structure for these visual questions designed to establish control groups. This structure enables us to conduct a quantitative analysis of the models' response tendencies, logical consistency, and various failure modes. In our evaluation on HallusionBench, we benchmarked 15 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V. Notably, all other evaluated models achieve accuracy below 16%. Moreover, our analysis not only highlights the observed failure modes, including language hallucination and visual illusion, but also deepens an understanding of these pitfalls. Our comprehensive case studies within HallusionBench shed light on the challenges of hallucination and illusion in LVLMs. Based on these insights, we suggest potential pathways for their future improvement. The benchmark and codebase can be accessed at https://github.com/tianyi-lab/HallusionBench.

Key facts. Cited by 572 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 66 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

44 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  2. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 42

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
30 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  9. 2023Instruction Tuning with GPT-4cited by 676
  10. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  11. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  12. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  13. 2018nocaps: novel object captioning at scalecited by 444
  14. 2023Larger language models do in-context learning differentlycited by 386
  15. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  16. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  17. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  18. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  19. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  20. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  21. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  22. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  23. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  24. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  25. 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
  26. 2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Mapscited by 35
  27. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33
  28. 2023COVID-VTS: Fact Extraction and Verification on Short Video Platformscited by 15
  29. 2023Module-wise Adaptive Distillation for Multimodality Foundation Modelscited by 12
  30. 2023DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documentscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 115; 2025: 206; 2026: 245.

572 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  8. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  9. 2025Qwen3-Omni Technical Reportcited by 451
  10. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  11. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  12. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
38 more
  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  3. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  4. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  5. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  6. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  7. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  8. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  9. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  10. 2026Qwen3.5-Omni Technical Reportcited by 146
  11. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  12. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  13. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  14. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  15. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  16. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  17. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  18. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  19. 2025Ovis2.5 Technical Reportcited by 103
  20. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  21. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  22. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  23. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  24. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  25. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  26. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  27. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  28. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  29. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  30. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  31. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  32. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  33. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  34. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  35. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  36. 2025Ovis-U1 Technical Reportcited by 65
  37. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  38. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62

50 most cited of 572 All 572 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation
  2. 2024Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models
  3. 2026HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models
  4. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  5. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediations
  6. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
  7. 2025Visual hallucination detection in large vision-language models via evidential conflict
  8. 2024H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models