# UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning

arXiv: https://arxiv.org/abs/2106.14019 · HTML: https://mlsanity.com/paper/2106.14019/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Discriminability objective for training descriptive captions](https://mlsanity.com/paper/1803.04376/) (2018)
- [Learning to Evaluate Image Captioning](https://mlsanity.com/paper/1806.06422/) (2018)
- [From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge](https://mlsanity.com/paper/1511.03292/) (2015)
- [Compare and Reweight: Distinctive Image Captioning Using Similar Images Sets](https://mlsanity.com/paper/2007.06877/) (2020)
- [VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions](https://mlsanity.com/paper/1907.09340/) (2019)
- [Quality Estimation for Image Captions Based on Large-scale Human Evaluations](https://mlsanity.com/paper/1909.03396/) (2019)

## Built on by (most cited first, 30 of 37)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [Fine-grained Image Captioning with CLIP Reward](https://mlsanity.com/paper/2205.13115/) (2022)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation](https://mlsanity.com/paper/2305.06002/) (2023)
- [Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric](https://mlsanity.com/paper/2402.06900/) (2024)
- [Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics](https://mlsanity.com/paper/2205.10646/) (2022)
- [Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives](https://mlsanity.com/paper/2503.14604/) (2025)
- [Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training](https://mlsanity.com/paper/2410.07336/) (2024)
- [BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues](https://mlsanity.com/paper/2407.20341/) (2024)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics](https://mlsanity.com/paper/2305.14998/) (2023)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction](https://mlsanity.com/paper/2402.17969/) (2024)
- [PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning](https://mlsanity.com/paper/2303.08389/) (2023)
- [VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions](https://mlsanity.com/paper/2509.25818/) (2025)
- [Cobra Effect in Reference-Free Image Captioning Metrics](https://mlsanity.com/paper/2402.11572/) (2024)
- [VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis](https://mlsanity.com/paper/2509.16538/) (2025)
- [Cross-Modal Similarity-Based Curriculum Learning for Image Captioning](https://mlsanity.com/paper/2212.07075/) (2022)
- [ContextRef: Evaluating Referenceless Metrics For Image Description Generation](https://mlsanity.com/paper/2309.11710/) (2023)
- [Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models](https://mlsanity.com/paper/2412.14613/) (2024)
- [Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics](https://mlsanity.com/paper/2412.10594/) (2024)
- [L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training](https://mlsanity.com/paper/2507.08710/) (2025)
- [DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioning](https://mlsanity.com/paper/2512.14420/) (2025)
- [Do Image-Text Metrics Respect Semantic Invariances?](https://mlsanity.com/paper/2605.24702/) (2026)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2606.29997/) (2026)
- [How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image](https://mlsanity.com/paper/2204.05533/) (2022)

## Abstract (arXiv)

Despite the success of various text generation metrics such as BERTScore, it is still difficult to evaluate the image captions without enough reference captions due to the diversity of the descriptions. In this paper, we introduce a new metric UMIC, an Unreferenced Metric for Image Captioning which does not require reference captions to evaluate image captions. Based on Vision-and-Language BERT, we train UMIC to discriminate negative captions via contrastive learning. Also, we observe critical problems of the previous benchmark dataset (i.e., human annotations) on image captioning metric, and introduce a new collection of human annotations on the generated captions. We validate UMIC on four datasets, including our new dataset, and show that UMIC has a higher correlation than all previous metrics that require multiple references. We release the benchmark dataset and pre-trained models to compute the UMIC.