mlsanityData to 7 Oct 2026

UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning

arXiv 2106.14019 ↗First posted 26 Jun 2021cs.CL cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

Despite the success of various text generation metrics such as BERTScore, it is still difficult to evaluate the image captions without enough reference captions due to the diversity of the descriptions. In this paper, we introduce a new metric UMIC, an Unreferenced Metric for Image Captioning which does not require reference captions to evaluate image captions. Based on Vision-and-Language BERT, we train UMIC to discriminate negative captions via contrastive learning. Also, we observe critical problems of the previous benchmark dataset (i.e., human annotations) on image captioning metric, and introduce a new collection of human annotations on the generated captions. We validate UMIC on four datasets, including our new dataset, and show that UMIC has a higher correlation than all previous metrics that require multiple references. We release the benchmark dataset and pre-trained models to compute the UMIC.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 26 Jun 2021.

Builds on

17 of 39 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2016Self-critical Sequence Training for Image Captioningcited by 544
  10. 2019Attention on Attention for Image Captioningcited by 289
  11. 2014From Captions to Visual Concepts and Backcited by 109
  12. 2018Discriminability objective for training descriptive captionscited by 67
5 more
  1. 2018Learning to Evaluate Image Captioningcited by 53
  2. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  3. 2020Compare and Reweight: Distinctive Image Captioning Using Similar Images Setscited by 26
  4. 2019VIFIDEL: Evaluating the Visual Fidelity of Image Descriptionscited by 21
  5. 2019Quality Estimation for Image Captions Based on Large-scale Human Evaluationscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 5; 2024: 10; 2025: 10; 2026: 5.

37 papers.

Sort:

By area:

  1. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  2. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  3. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  4. 2022Image Difference Captioning with Pre-training and Contrastive Learningcited by 39
  5. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  6. 2024Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metriccited by 23
  7. 2022Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metricscited by 18
  8. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 17
  9. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  10. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  11. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  12. 2023An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metricscited by 11
25 more
  1. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  2. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  3. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7
  4. 2023PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioningcited by 5
  5. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  6. 2024Cobra Effect in Reference-Free Image Captioning Metricscited by 4
  7. 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysiscited by 4
  8. 2022Cross-Modal Similarity-Based Curriculum Learning for Image Captioningcited by 3
  9. 2023ContextRef: Evaluating Referenceless Metrics For Image Description Generationcited by 3
  10. 2024Towards Unified Benchmark and Models for Multi-Modal Perceptual Metricscited by 2
  11. 2024Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Modelscited by 2
  12. 2025L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Trainingcited by 2
  13. 2025DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioningcited by 2
  14. 2021Goal-driven text descriptions for imagescited by 1
  15. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 1
  16. 2022How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Imagecited by 0
  17. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  18. 2025Group-based Distinctive Image Captioning with Memory Difference Encoding and Attentioncited by 0
  19. 2025Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulationcited by 0
  20. 2025Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clusteringcited by 0
  21. 2025Generating Accurate and Detailed Captions for High-Resolution Imagescited by 0
  22. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMscited by 0
  23. 2026Evaluating Remote Sensing Image Captions Beyond Metric Biasescited by 0
  24. 2026MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluationcited by 0
  25. 2026Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluationcited by 0