mlsanityData to 7 Oct 2026

DASH: Detection and Assessment of Systematic Hallucinations of VLMs

arXiv 2503.23573 ↗First posted 30 Mar 2025cs.CV cs.AI cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-language models (VLMs) are prone to object hallucinations, where they erroneously indicate the presenceof certain objects in an image. Existing benchmarks quantify hallucinations using relatively small, labeled datasets. However, this approach is i) insufficient to assess hallucinations that arise in open-world settings, where VLMs are widely used, and ii) inadequate for detecting systematic errors in VLMs. We propose DASH (Detection and Assessment of Systematic Hallucinations), an automatic, large-scale pipeline designed to identify systematic hallucinations of VLMs on real-world images in an open-world setting. A key component is DASH-OPT for image-based retrieval, where we optimize over the ''natural image manifold'' to generate images that mislead the VLM. The output of DASH consists of clusters of real and semantically similar images for which the VLM hallucinates an object. We apply DASH to PaliGemma and two LLaVA-NeXT models across 380 object classes and, in total, find more than 19k clusters with 950k images. We study the transfer of the identified systematic hallucinations to other VLMs and show that fine-tuning PaliGemma with the model-specific images obtained with DASH mitigates object hallucinations. Code and data are available at https://YanNeu.github.io/DASH.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 61 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

44 of 61 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning22 Jul
  2. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models24 Jun
  3. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models8 May
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  8. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  9. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  10. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
22 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  10. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  11. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  12. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  13. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  14. 2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 153
  15. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  16. 2023Generating images of rare concepts using pre-trained diffusion modelscited by 78
  17. 2023On the De-duplication of LAION-2Bcited by 47
  18. 2023Norm-guided latent space exploration for text-to-image generationcited by 43
  19. 2022Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNetcited by 41
  20. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  21. 2023Identification of Systematic Errors of Image Classifiers on Rare Subgroupscited by 17
  22. 2023DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanationscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025A Survey on Efficient Vision-Language Modelscited by 19
  2. 2025RePOPE: Impact of Annotation Errors on the POPE Benchmarkcited by 9
  3. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  4. 2025Measuring Epistemic Humility in Multimodal Large Language Modelscited by 4
  5. 2025GHOST: Hallucination-Inducing Image Generation for Multimodal LLMscited by 4
  6. 2026SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cuescited by 4
  7. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
  8. 2025Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentationcited by 2
  9. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  10. 2026ZAYA1-VL-8B Technical Reportcited by 2
  11. 2026PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoningcited by 1
  12. 2026GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Modelscited by 0
2 more
  1. 2026Zamba2-VL Technical Reportcited by 0
  2. 2026Detecting Object Hallucinations in Large Vision-Language Models via Cross-Modal Attention Drifts and Mask-Based Verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  2. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  3. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs
  4. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  5. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models
  6. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
  7. 2025Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention
  8. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models