mlsanityData to 7 Oct 2026

HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioning

arXiv 2503.00436 ↗First posted 1 Mar 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In the dynamic landscape of artificial intelligence, the exploration of hallucinations within vision-language (VL) models emerges as a critical frontier. This work delves into the intricacies of hallucinatory phenomena exhibited by widely used image captioners, unraveling interesting patterns. Specifically, we step upon previously introduced techniques of conceptual counterfactual explanations to address VL hallucinations. The deterministic and efficient nature of the employed conceptual counterfactuals backbone is able to suggest semantically minimal edits driven by hierarchical knowledge, so that the transition from a hallucinated caption to a non-hallucinated one is performed in a black-box manner. HalCECE, our proposed hallucination detection framework is highly interpretable, by providing semantically meaningful edits apart from standalone numbers, while the hierarchical decomposition of hallucinated concepts leads to a thorough hallucination analysis. Another novelty tied to the current work is the investigation of role hallucinations, being one of the first works to involve interconnections between visual concepts in hallucination detection. Overall, HalCECE recommends an explainable direction to the crucial field of VL hallucination detection, thus fostering trustworthy evaluation of current and future VL systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

37 of 51 references.

References · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
25 more
  1. 2018Object Hallucination in Image Captioningcited by 661
  2. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  5. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  8. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  9. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  10. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  11. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  12. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  13. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  14. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  15. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  16. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 44
  17. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  18. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28
  19. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 27
  20. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27
  21. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)cited by 23
  22. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
  23. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similaritiescited by 7
  24. 2024Structure Your Data: Towards Semantic Graph Counterfactualscited by 6
  25. 2022Towards explainable evaluation of language models on the semantic similarity of visual conceptscited by 5

Built on by

Not cited yet.