Are metrics measuring what they should? An evaluation of image captioning task metrics
Not on the map. Why?
Abstract (arXiv)
Image Captioning is a current research task to describe the image content using the objects and their relationships in the scene. To tackle this task, two important research areas converge, artificial vision, and natural language processing. In Image Captioning, as in any computational intelligence task, the performance metrics are crucial for knowing how well (or bad) a method performs. In recent years, it has been observed that classical metrics based on n-grams are insufficient to capture the semantics and the critical meaning to describe the content in an image. Looking to measure how well or not the set of current and more recent metrics are doing, in this article, we present an evaluation of several kinds of Image Captioning metrics and a comparison between them using the well-known MS COCO dataset. The metrics were selected from the most used in prior works, they are those based on n-grams as BLEU, SacreBLEU, METEOR, ROGUE-L, CIDEr, SPICE, and those based on embeddings, such as BERTScore and CLIPScore. For this, we designed two scenarios; 1) a set of artificially build captions with several qualities, and 2) a comparison of some state-of-the-art Image Captioning methods. Interesting findings were found trying to answer the questions: Are the current metrics helping to produce high-quality captions? How do actual metrics compare to each other? What are the metrics really measuring?
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 59 of its 86 references that are in the map. First posted on arXiv 4 Jul 2022.
Builds on
59 of 86 references.
References · 59
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
38 more
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2014Show and Tell: A Neural Image Caption Generatorcited by 877
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2015Sequence Level Training with Recurrent Neural Networkscited by 568
- 2016Self-critical Sequence Training for Image Captioningcited by 544
- 2019Release Strategies and the Social Impacts of Language Modelscited by 505
- 2018Stacked Cross Attention for Image-Text Matchingcited by 414
- 2019How to Fine-Tune BERT for Text Classification?cited by 409
- 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
- 2019Meshed-Memory Transformer for Image Captioningcited by 343
- 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
- 2019Attention on Attention for Image Captioningcited by 289
- 2016Image Captioning with Semantic Attentioncited by 260
- 2017Why We Need New Evaluation Metrics for NLGcited by 232
- 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
- 2020X-Linear Attention Networks for Image Captioningcited by 198
- 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
- 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
- 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
- 2014From Captions to Visual Concepts and Backcited by 109
- 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
- 2020Visual Commonsense R-CNNcited by 106
- 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
- 2019Polysemous Visual-Semantic Embedding for Cross-Modal Retrievalcited by 83
- 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
- 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
- 2021CPTR: Full Transformer Network for Image Captioningcited by 55
- 2020Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusioncited by 53
- 2017Convolutional Image Captioningcited by 52
- 2016Re-evaluating Automatic Metrics for Image Captioningcited by 45
- 2017Deep Reinforcement Learning-based Image Captioning with Embedding Rewardcited by 39
- 2016Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measurescited by 36
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.