# VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions

arXiv: https://arxiv.org/abs/1907.09340 · HTML: https://mlsanity.com/paper/1907.09340/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2019.

## Built on by (most cited first, 21 of 21)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning](https://mlsanity.com/paper/2106.14019/) (2021)
- [ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models](https://mlsanity.com/paper/2311.07022/) (2023)
- [Mitigating Open-Vocabulary Caption Hallucinations](https://mlsanity.com/paper/2312.03631/) (2023)
- [A Survey on Quality Metrics for Text-to-Image Generation](https://mlsanity.com/paper/2403.11821/) (2024)
- [COSMic: A Coherence-Aware Generation Metric for Image Descriptions](https://mlsanity.com/paper/2109.05281/) (2021)
- [Text-to-Image Synthesis: A Decade Survey](https://mlsanity.com/paper/2411.16164/) (2024)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells](https://mlsanity.com/paper/2409.19801/) (2024)
- [Multimodal Data Augmentation for Image Captioning using Diffusion Models](https://mlsanity.com/paper/2305.01855/) (2023)
- [PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning](https://mlsanity.com/paper/2303.08389/) (2023)
- [Paraphrasing Is All You Need for Novel Object Captioning](https://mlsanity.com/paper/2209.12343/) (2022)
- [VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis](https://mlsanity.com/paper/2509.16538/) (2025)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Contrastive Semantic Similarity Learning for Image Captioning Evaluation with Intrinsic Auto-encoder](https://mlsanity.com/paper/2106.15312/) (2021)
- [ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation](https://mlsanity.com/paper/2403.01306/) (2024)
- [Disentangling Fine-Tuning from Pre-Training in Visual Captioning with Hybrid Markov Logic](https://mlsanity.com/paper/2503.13847/) (2025)
- [On Explaining Visual Captioning with Hybrid Markov Logic Networks](https://mlsanity.com/paper/2507.21246/) (2025)
- [Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations](https://mlsanity.com/paper/2509.25844/) (2025)

## Abstract (arXiv)

We address the task of evaluating image description generation systems. We propose a novel image-aware metric for this task: VIFIDEL. It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description. The metric is also able to take into account the relative importance of objects mentioned in human reference descriptions during evaluation. Even if these human reference descriptions are not available, VIFIDEL can still reliably evaluate system descriptions. The metric achieves high correlation with human judgments on two well-known datasets and is competitive with metrics that depend on human references