# Learning to Evaluate Image Captioning

arXiv: https://arxiv.org/abs/1806.06422 · HTML: https://mlsanity.com/paper/1806.06422/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2018.

## Built on by (most cited first, 30 of 53)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [CLAIR: Evaluating Image Captions with Large Language Models](https://mlsanity.com/paper/2310.12971/) (2023)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [Automatic Text Evaluation through the Lens of Wasserstein Barycenters](https://mlsanity.com/paper/2108.12463/) (2021)
- [UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning](https://mlsanity.com/paper/2106.14019/) (2021)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Mutual Information Divergence: A Unified Metric for Multimodal Generative Models](https://mlsanity.com/paper/2205.13445/) (2022)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy](https://mlsanity.com/paper/2402.07270/) (2024)
- [Neptune: The Long Orbit to Benchmarking Long Video Understanding](https://mlsanity.com/paper/2412.09582/) (2024)
- [G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o](https://mlsanity.com/paper/2412.13647/) (2024)
- [MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration](https://mlsanity.com/paper/2204.08058/) (2022)
- [Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?](https://mlsanity.com/paper/2404.06510/) (2024)
- [Affection: Learning Affective Explanations for Real-World Visual Data](https://mlsanity.com/paper/2210.01946/) (2022)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis](https://mlsanity.com/paper/2210.05035/) (2022)
- [A Survey on Quality Metrics for Text-to-Image Generation](https://mlsanity.com/paper/2403.11821/) (2024)
- [Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training](https://mlsanity.com/paper/2410.07336/) (2024)
- [CaptionQA: Is Your Caption as Useful as the Image Itself?](https://mlsanity.com/paper/2511.21025/) (2025)
- [Test-Time Distribution Normalization for Contrastively Learned Vision-language Models](https://mlsanity.com/paper/2302.11084/) (2023)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)
- [Dynamic population-based meta-learning for multi-agent communication with natural language](https://mlsanity.com/paper/2110.14241/) (2021)
- [COSMic: A Coherence-Aware Generation Metric for Image Descriptions](https://mlsanity.com/paper/2109.05281/) (2021)

## Abstract (arXiv)

Evaluation metrics for image captioning face two challenges. Firstly, commonly used metrics such as CIDEr, METEOR, ROUGE and BLEU often do not correlate well with human judgments. Secondly, each metric has well known blind spots to pathological caption constructions, and rule-based metrics lack provisions to repair such blind spots once identified. For example, the newly proposed SPICE correlates well with human judgments, but fails to capture the syntactic structure of a sentence. To address these two challenges, we propose a novel learning based discriminative evaluation metric that is directly trained to distinguish between human and machine-generated captions. In addition, we further propose a data augmentation scheme to explicitly incorporate pathological transformations as negative examples during training. The proposed metric is evaluated with three kinds of robustness tests and its correlation with human judgments. Extensive experiments show that the proposed data augmentation scheme not only makes our metric more robust toward several pathological transformations, but also improves its correlation with human judgments. Our metric outperforms other metrics on both caption level human correlation in Flickr 8k and system level human correlation in COCO. The proposed approach could be served as a learning based evaluation metric that is complementary to existing rule-based metrics.