# InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation

arXiv: https://arxiv.org/abs/2305.06002 · HTML: https://mlsanity.com/paper/2305.06002/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video](https://mlsanity.com/paper/2302.00402/) (2023)
- [E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning](https://mlsanity.com/paper/2106.01804/) (2021)
- [TIGEr: Text-to-Image Grounding for Image Caption Evaluation](https://mlsanity.com/paper/1909.02050/) (2019)
- [XGPT: Cross-modal Generative Pre-Training for Image Captioning](https://mlsanity.com/paper/2003.01473/) (2020)
- [Transparent Human Evaluation for Image Captioning](https://mlsanity.com/paper/2111.08940/) (2021)
- [UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning](https://mlsanity.com/paper/2106.14019/) (2021)
- [ICECAP: Information Concentrated Entity-aware Image Captioning](https://mlsanity.com/paper/2108.02050/) (2021)
- [Question-controlled Text-aware Image Captioning](https://mlsanity.com/paper/2108.02059/) (2021)
- [REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning](https://mlsanity.com/paper/1909.02217/) (2019)

## Built on by (most cited first, 25 of 25)

- [Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models](https://mlsanity.com/paper/2409.10695/) (2024)
- [FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model](https://mlsanity.com/paper/2406.06004/) (2024)
- [AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering](https://mlsanity.com/paper/2311.14906/) (2023)
- [UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Prediction](https://mlsanity.com/paper/2403.16831/) (2024)
- [Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives](https://mlsanity.com/paper/2503.14604/) (2025)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics](https://mlsanity.com/paper/2305.14998/) (2023)
- [VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation](https://mlsanity.com/paper/2505.23484/) (2025)
- [CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells](https://mlsanity.com/paper/2409.19801/) (2024)
- [I0T: Embedding Standardization Method Towards Zero Modality Gap](https://mlsanity.com/paper/2412.14384/) (2024)
- [Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction](https://mlsanity.com/paper/2402.17969/) (2024)
- [EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations](https://mlsanity.com/paper/2506.24016/) (2025)
- [Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies](https://mlsanity.com/paper/2410.08860/) (2024)
- [Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?](https://mlsanity.com/paper/2502.06600/) (2025)
- [Cobra Effect in Reference-Free Image Captioning Metrics](https://mlsanity.com/paper/2402.11572/) (2024)
- [Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models](https://mlsanity.com/paper/2412.14613/) (2024)
- [DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioning](https://mlsanity.com/paper/2512.14420/) (2025)
- [Do Image-Text Metrics Respect Semantic Invariances?](https://mlsanity.com/paper/2605.24702/) (2026)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Unifying Vision-Language Latents for Zero-label Image Caption Enhancement](https://mlsanity.com/paper/2510.12931/) (2025)
- [A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates](https://mlsanity.com/paper/2504.01225/) (2025)
- [VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment](https://mlsanity.com/paper/2509.21609/) (2025)
- [MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation](https://mlsanity.com/paper/2605.06080/) (2026)
- [BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model](https://mlsanity.com/paper/2605.21728/) (2026)
- [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](https://mlsanity.com/paper/2609.09973/) (2026)

## Abstract (arXiv)

Automatic image captioning evaluation is critical for benchmarking and promoting advances in image captioning research. Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative. Instead, we humans can easily identify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality. To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC). Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level. The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than existing metrics on multiple benchmarks. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation. Our code and datasets are publicly available at https://github.com/HAWLYQ/InfoMetIC.