# Re-evaluating Automatic Metrics for Image Captioning

arXiv: https://arxiv.org/abs/1612.07600 · HTML: https://mlsanity.com/paper/1612.07600/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2016.

## Built on by (most cited first, 30 of 45)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions](https://mlsanity.com/paper/2406.07502/) (2024)
- [Transparent Human Evaluation for Image Captioning](https://mlsanity.com/paper/2111.08940/) (2021)
- [Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?](https://mlsanity.com/paper/2404.06510/) (2024)
- [Affection: Learning Affective Explanations for Real-World Visual Data](https://mlsanity.com/paper/2210.01946/) (2022)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [LineCap: Line Charts for Data Visualization Captioning Models](https://mlsanity.com/paper/2207.07243/) (2022)
- [CaptionQA: Is Your Caption as Useful as the Image Itself?](https://mlsanity.com/paper/2511.21025/) (2025)
- [Code Search based on Context-aware Code Translation](https://mlsanity.com/paper/2202.08029/) (2022)
- [Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching](https://mlsanity.com/paper/2110.02623/) (2021)
- [Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers](https://mlsanity.com/paper/2212.13196/) (2022)
- [Visually-Aware Context Modeling for News Image Captioning](https://mlsanity.com/paper/2308.08325/) (2023)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception](https://mlsanity.com/paper/2504.06666/) (2025)
- [Re-evaluating Word Mover's Distance](https://mlsanity.com/paper/2105.14403/) (2021)
- [Trust It or Not: Confidence-Guided Automatic Radiology Report Generation](https://mlsanity.com/paper/2106.10887/) (2021)
- [Going for GOAL: A Resource for Grounded Football Commentaries](https://mlsanity.com/paper/2211.04534/) (2022)
- [Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports](https://mlsanity.com/paper/2401.16578/) (2024)
- [Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?](https://mlsanity.com/paper/2502.04718/) (2025)
- [Diagnostic Captioning: A Survey](https://mlsanity.com/paper/2101.07299/) (2021)
- [Meta-evaluation of Conversational Search Evaluation Metrics](https://mlsanity.com/paper/2104.13453/) (2021)
- [JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models](https://mlsanity.com/paper/2311.04192/) (2023)
- [The Critique of Critique](https://mlsanity.com/paper/2401.04518/) (2024)
- [Model Fusion through Bayesian Optimization in Language Model Fine-Tuning](https://mlsanity.com/paper/2411.06710/) (2024)
- [Social Biases in Automatic Evaluation Metrics for NLG](https://mlsanity.com/paper/2210.08859/) (2022)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)

## Abstract (arXiv)

The task of generating natural language descriptions from images has received a lot of attention in recent years. Consequently, it is becoming increasingly important to evaluate such image captioning approaches in an automatic manner. In this paper, we provide an in-depth evaluation of the existing image captioning metrics through a series of carefully designed experiments. Moreover, we explore the utilization of the recently proposed Word Mover's Distance (WMD) document metric for the purpose of image captioning. Our findings outline the differences and/or similarities between metrics and their relative robustness by means of extensive correlation, accuracy and distraction based evaluations. Our results also demonstrate that WMD provides strong advantages over other metrics.