mlsanityData to 7 Oct 2026

Re-evaluating Automatic Metrics for Image Captioning

arXiv 1612.07600 ↗First posted 22 Dec 2016cs.CL cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

The task of generating natural language descriptions from images has received a lot of attention in recent years. Consequently, it is becoming increasingly important to evaluate such image captioning approaches in an automatic manner. In this paper, we provide an in-depth evaluation of the existing image captioning metrics through a series of carefully designed experiments. Moreover, we explore the utilization of the recently proposed Word Mover's Distance (WMD) document metric for the purpose of image captioning. Our findings outline the differences and/or similarities between metrics and their relative robustness by means of extensive correlation, accuracy and distraction based evaluations. Our results also demonstrate that WMD provides strong advantages over other metrics.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 9; 2023: 2; 2024: 7; 2025: 5; 2026: 6.

45 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  4. 2021ArtEmis: Affective Language for Visual Artcited by 103
  5. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  6. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  7. 2021Transparent Human Evaluation for Image Captioningcited by 40
  8. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  9. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  10. 2021SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysiscited by 18
  11. 2022LineCap: Line Charts for Data Visualization Captioning Modelscited by 17
  12. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 14
33 more
  1. 2021Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matchingcited by 13
  2. 2022Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformerscited by 13
  3. 2023Visually-Aware Context Modeling for News Image Captioningcited by 12
  4. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  5. 2025Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perceptioncited by 12
  6. 2022Code Search based on Context-aware Code Translationcited by 11
  7. 2021Re-evaluating Word Mover's Distancecited by 10
  8. 2021Trust It or Not: Confidence-Guided Automatic Radiology Report Generationcited by 9
  9. 2022Going for GOAL: A Resource for Grounded Football Commentariescited by 7
  10. 2024Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reportscited by 7
  11. 2021Diagnostic Captioning: A Surveycited by 6
  12. 2021Meta-evaluation of Conversational Search Evaluation Metricscited by 6
  13. 2023JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Modelscited by 6
  14. 2024The Critique of Critiquecited by 5
  15. 2024Model Fusion through Bayesian Optimization in Language Model Fine-Tuningcited by 5
  16. 2025Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?cited by 4
  17. 2022Social Biases in Automatic Evaluation Metrics for NLGcited by 3
  18. 2022Are metrics measuring what they should? An evaluation of image captioning task metricscited by 2
  19. 2022Predicting Word Learning in Children from the Performance of Computer Vision Systemscited by 2
  20. 2021The Role of the Input in Natural Language Video Descriptioncited by 1
  21. 2021Caption Generation on Scenes with Seen and Unseen Object Categoriescited by 1
  22. 2021Goal-driven text descriptions for imagescited by 1
  23. 2022Generative Pre-Trained Transformers for Biologically Inspired Designcited by 1
  24. 2025LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learningcited by 1
  25. 2021Empirical Analysis of Image Caption Generation using Deep Learningcited by 0
  26. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  27. 2025HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioningcited by 0
  28. 2026Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Datasetcited by 0
  29. 2026Grammatical Error Correction Evaluation by Optimally Transporting Edit Representationcited by 0
  30. 2026Barycentric alignment for instance-level comparison of neural representationscited by 0
  31. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  32. 2026Search-based Testing of Vision Language Models for In-Car Scene Understandingcited by 0
  33. 2026Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Modelscited by 0