mlsanityData to 14 Oct 2026

ZINA: Multimodal Fine-grained Hallucination Detection and Editing

arXiv 2506.13130 ↗First posted 16 Jun 2025cs.CV cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) often generate hallucinations, where the output deviates from the visual content. Given that these hallucinations can take diverse forms, detecting hallucinations at a fine-grained level is essential for comprehensive evaluation and analysis. To this end, we propose a novel task of multimodal fine-grained hallucination detection and editing for MLLMs. Moreover, we propose ZINA, a novel method that identifies hallucinated spans at a fine-grained level, classifies their error types into six categories, and suggests appropriate refinements. To train and evaluate models for this task, we construct VisionHall, a dataset comprising 6.9k outputs from twelve MLLMs manually annotated by 211 annotators, and 20k synthetic samples generated using a graph-based method that captures dependencies among error types. We demonstrated that ZINA outperformed existing methods, including GPT-4o and Llama-3.2, in both detection and editing tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 58 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

49 of 58 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025HalLoc: Token-level Localization of Hallucinations for Vision Language Models12 Jun
  2. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning8 Jun
  3. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models30 Jun
  4. 2024ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models24 Mar
  5. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  6. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  7. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models3 Oct
  8. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  9. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
28 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  12. 2018Object Hallucination in Image Captioningcited by 669
  13. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  14. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  15. 2018nocaps: novel object captioning at scalecited by 444
  16. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  17. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  18. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  19. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  20. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
  21. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  22. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
  23. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  24. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
  25. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  26. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  27. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  28. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  2. 2026GAVEL: Grounded Caption Error Verification and Localizationcited by 1
  3. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0
  4. 2026UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representationscited by 0
  5. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
  2. 2025A Survey of Multimodal Hallucination Evaluation and Detection
  3. 2026Scalable Token-Level Hallucination Detection in Large Language Models
  4. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
  5. 2026OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
  6. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
  7. 2024Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data
  8. 2025HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detection