mlsanityData to 7 Oct 2026

FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models

arXiv 2311.01477 ↗First posted 2 Nov 2023cs.CVcited by 44

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

We introduce FaithScore (Faithfulness to Atomic Image Facts Score), a reference-free and fine-grained evaluation metric that measures the faithfulness of the generated free-form answers from large vision-language models (LVLMs). The FaithScore evaluation first identifies sub-sentences containing descriptive statements that need to be verified, then extracts a comprehensive list of atomic facts from these sub-sentences, and finally conducts consistency verification between fine-grained atomic facts and the input image. Meta-evaluation demonstrates that our metric highly correlates with human judgments of faithfulness. We collect two benchmark datasets (i.e. LLaVA-1k and MSCOCO-Cap) for evaluating LVLMs instruction-following hallucinations. We measure hallucinations in state-of-the-art LVLMs with FaithScore on the datasets. Results reveal that current systems are prone to generate hallucinated content unfaithful to the image, which leaves room for future improvements. We hope our metric FaithScore can help evaluate future LVLMs in terms of faithfulness and provide insightful advice for enhancing LVLMs' faithfulness.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 46 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

26 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
13 more
  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  5. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  6. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  7. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  8. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  9. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  10. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  11. 2016Visual Storytellingcited by 176
  12. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  13. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 18; 2026: 16.

44 papers.

Sort:

By area:

  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  2. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  3. 2024Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucinationcited by 29
  4. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  5. 2024Reference-free Hallucination Detection for Large Vision-Language Modelscited by 23
  6. 2024Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Trainingcited by 17
  7. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  8. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 12
  9. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  10. 2025Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Modelscited by 9
  11. 2024Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Modelscited by 8
  12. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
32 more
  1. 2024Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuningcited by 7
  2. 2025Faithful-First Reasoning, Planning, and Acting for Multimodal LLMscited by 7
  3. 2024AIGCs Confuse AI Too: Investigating and Explaining Synthetic Image-induced Hallucinations in Large Vision-Language Modelscited by 6
  4. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMscited by 4
  5. 2024Detailed Object Description with Controllable Dimensionscited by 3
  6. 2025Improving Alignment in LVLMs with Debiased Self-Judgmentcited by 3
  7. 2025Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicinecited by 2
  8. 2025SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revisioncited by 2
  9. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  10. 2025A Study on the Framework for Evaluating the Ethics and Trustworthiness of Generative AIcited by 2
  11. 2026ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Bettercited by 2
  12. 2026BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioningcited by 2
  13. 2025Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -cited by 1
  14. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  15. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  16. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  17. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  18. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  19. 2026RSHallu: Dual-Mode Hallucination Evaluation for Remote-Sensing Multimodal Large Language Models with Domain-Tailored Mitigationcited by 1
  20. 2026Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Modelscited by 1
  21. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  22. 2026Failing to See or Failing to Know? Attributing Errors in Vision-Language Modelscited by 1
  23. 2025Iterative Tree Analysis for Medical Criticscited by 0
  24. 2025Critique Before Thinking: Mitigating Hallucination through Rationale-Augmented Instruction Tuningcited by 0
  25. 2026Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertaintycited by 0
  26. 2026Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactionscited by 0
  27. 2026Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQAcited by 0
  28. 2026Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Studycited by 0
  29. 2026Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognitioncited by 0
  30. 2026Program-Verified Self-Evolution for Vision-Language Modelscited by 0
  31. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Modelscited by 0
  32. 2026A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
  2. 2025SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs
  3. 2024H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models
  4. 2025Taming Object Hallucinations with Verified Atomic Confidence Estimation
  5. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models
  6. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention
  7. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context
  8. 2026ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models